I (along with David Nalley - ke4qqq) presented a tutorial on Cloud Networking for cloud operators (not users) at LISA 2012

1. Networking in the Cloud Age!
With references to Apache CloudStack!
!
December 11 2012!
!
Chiradeep Vittal!
@chiradeep!
David Nalley!
@ke4qqq!

2. Agenda!
• Why virtual networks?!
• Basic principles of Cloud Networking!
• Service insertion in virtual networks!
• Virtual Networking using L3 isolation!
• Networking in Apache CloudStack!
• Software Defined Networking!
• Wrap-up!

3. Apache CloudStack!
• Secure, multi-tenant cloud
orchestration platform!
– Turnkey platform for delivering
IaaS clouds!
– Over 150 commercial
Build your cloud the way deployments: private and public!
the world’s most successful – Full featured GUI, end-user API
clouds are built!
and admin API!

4. Apache CloudStack!
• Open Source!
• Apache License!
• Incubating in the Apache
Software Foundation since
April 2012!
Build your cloud the way
the world’s most successful • Open Source since May
clouds are built! 2010!
• In production since 2009!

5. Networking in the Cloud Age!
DRIVERS!

6. Drivers!
New-style!
IAAS! Workload!
Agility! Application owns availability!
Virtualization!
API! High bandwidth!
Self-service!
Elasticity!
Scale! Low cost! Distributed!
L3! Cookie cutter!
Multi-tenancy!
These classes of drivers (IAAS and new-style workloads) are highly
complementary and therefore most new-style applications operate on IAAS!

7. Traditional Style!
Traditional!
IAAS! Workload!
Agility!
Infra owns availability!
Virtualization!
API!
Complex Packet Filters!
Elasticity! Scale! High cost!
Self-service!
Gold-plated!
Multi-tenancy! Infra!
L2!
It is possible to realize some of the benefits of IAAS for traditional workloads !

8. Traditional infra can be IAAS!
IAAS! Agility!
Gold-plated!
Virtualization!
Infra!
API!
Infra owns availability!
Elasticity! Scale! High cost!
Self-service!
Multi-tenancy! L2!
Complex Packet Filters!
It is possible to realize some of the benefits of IAAS for traditional infrastructure!

9. Traditional! Cloud!
• 10x more
scaleable!
• 2-5x lower
cost!
• 100% more
open!
Built for traditional Designed around big data,
enterprise apps & client- massive scale & next-gen
server compute! apps!
• Enterprise arch for 100s of • Cloud architecture for 1000s
hosts! of hosts!
• Scale-up (server clusters) ! • Scale-out (multi-site server
• Apps assume reliability! farms)!
• IT Mgmt-centric [1:Dozens]! • Apps assume failure!
• Proprietary vendor stack! • Autonomic [1:1,000’s]!
• Open, value-added stack!

10. Defining Cloud Computing (IAAS)!
• Agility!
– Re-provision complex infrastructure topologies in
minutes, not days
• API!
– Automate complex infrastructure tasks
• Virtualization!
– Enables workload mobility and load sharing
• Multi-tenancy!
– Share resources and costs

11. Defining Cloud Computing (IAAS)!
• Scalability!
– Ability to consume resources limited by budget, not
by infrastructure
• Elasticity!
– Scale up and down on demand
– Reduce need to engineer for peak load
• Self-service!
– No IT assistance!

12. Cloud Networking
Requirements!
• Agile!
– Complex networking topologies created by non-
network engineers
• API!
– Language to talk with the network infrastructure
layer (not CLI)
• Virtualization!
– Hypervisor-level switches work together with physical
infrastructure

13. Cloud Networking
Requirements!
• Scalability!
– Usually means L3 in the physical infrastructure
• Elasticity!
– Release resources when not in use
– Introduce new resources on demand
• Self-service!
– Novices deploying, maintaining, troubleshooting
virtual networks

14. Cloud-Style Workloads!
• Low cost!
– Standardized, cookie cutter infrastructure
– Highly automated and efficient
• L3!
– Applications do not need persistent ip/mac
– L2 adjacency not required
• Application owns availability!
– At scale everything breaks
– Focus on MTTR instead of MTBF

15. Scale!
“At scale, everything breaks”!
-­‐
Urs
Hölzle,
Google!
" " "!
Server failure comes from:!
ᵒ 70% - hard disk!
8%
ᵒ 6% - RAID controller!
ᵒ 5% - memory!
ᵒ 18% - other factors!
Application can still fail for
Annual
Failure
Rate
of
servers
other reasons:!
ᵒ Network failure!
Kashi
Venkatesh
Vishwanath
and
Nachiappan
Nagappan,
Characterizing
ᵒ Software bugs!
Cloud
Compu3ng
Hardware
Reliability,
ᵒ Human admin error!
SoCC’10

16. Redundancy helps a little!
• Bugs in failover
40%!
mechanism!
• Incorrect configuration!
• Protocol issues such
as TCP back-off,
timeouts, and
Effectiveness of network
redundancy in reducing spanning tree
failures! reconfiguration!
Phillipa Gill, Navendu Jain &
Nachiappan Nagappan, Understanding
Network Failures in Data Centers:
Measurement, Analysis and
Implications, SIGCOMM 2011 !
16!

17. Reliability Strategies!
Cloud workloads!
Traditional-Style! New (“Amazon”) Style!
Reliable hardware, backup Tell users to expect failure.
entire cloud, and restore for Users to build apps that can
users when failure happens! withstand infrastructure
failure!
Both styles of workloads must run reliably in the cloud!

18. Reliability Styles!
Traditional workload! Cloud workload!
Link aggregation! VM backup/snapshots !
Storage multi-pathing! Ephemeral resources!
VM HA, fault tolerance! Chaos monkey!
VM live migration! Multi-site redundancy!
Expect reliability. Back-up entire Expect failure. Design app for failure.
cloud. Admin controlled failure Self-service failure handling!
handling! Think Amazon Web Services!
Think Server Virtualization!

19. Traditional Enterprise network!
Backbone/
Internet!
Core Routers!
N-S traffic!
…! Access Routers!
Packet Filters!
Aggregation Switches!
Load Balancers!
…! Top of Rack Switches!
Servers!

20. Enterprise networks!
• Hierarchical tree structure!
– Assumes N-S traffic predominant
• L2 domains!
– Susceptible to flooding
– Wasted capacity due to STP
• Services provided by redundant HW appliances!
– Firewall, IDS, ACL, Loadbalancer
– Often need L2 adjacency!
• Complex engineering, limited scale!

21. Scaled out network!
Backbone/
Internet!
Spine Routers!
Leaf Routers!
…! Servers!
Host-based! Server Load Balancing!
firewalls and ACL!

22. Scaled out network!
• L3 (routed) network!
– ECMP for increased bandwidth/redundancy
• No oversubscription!
– Uniform access to bandwidth
• Predominantly east-west traffic!
• Commodity hardware!
• Services provided at the host / vm level!
– Firewall, IDS, load balancing.

23. Networking in the Cloud Age!
VIRTUAL NETWORKING
PRINCIPLES!

24. The illusion of isolated networks on top of
shared physical infrastructure!

25. Usually requires!
• Hypervisors!
– To share the same host with multiple tenants
• Virtual (software) switches!
– Port-level control to provide isolation
• Services provided in software / virtual contexts!
– Loadbalancer / firewall virtual appliances
– Host-based firewalls

26. Virtual-to-Physical Mapping!
• Option 1: VLAN!
– 1 virtual network = 1 VLAN in physical infra
!
• Option 2: Tunnels!
– VxLAN!
– (NV) GRE!
– STT!
– Others: MAC-in-MAC, NVO3, MPLS!

27. Virtual-to-Physical Mapping!
• Option 3: IP address re-write!
– 1 tenant address mapped to 1 different provider
address
– Hyper-V only (possible with KVM/Xen)
• Option 4: No mapping !
– Tenant address is present on physical network
– Tenants isolated from each other and physical
network using packet filters in hypervisor
– L3 isolation is CloudStack’s term for this mode
– Also called “Basic Networking”.

28. Virtual Switches!
• Linux bridge!
– KVM, XenServer, XCP, Oracle VM
• Open vSwitch (OVS)!
– KVM, XenServer, XCP
• VMware options!
– vSphere
– Distributed vSwitch (DVS)
– Cisco Nexus 1000v

29. Virtual Switches!
Hypervisor Host!
VM A1! VM A2! VM B1! VM C1!
untagged (usually)!
Virtual Nics!
vswitch! vswitch! vswitch!
Physical !
Nics!
192.168.1.0/24! VLAN TRUNK!
VLAN 10!
192.168.1.0/24!
VLAN 20!
10.1.1.0/24! VLAN 30!

30. Egress Traffic from VM!
Ethernet frame from VM A1 to vswitch (untagged)
Payload (IP Packet)
06:00:01:AA:BB:CC
06:02:12:1D:1E
0x800
46-1500 octets
Dest, addr
Src, addr
Type
Ethernet frame from vswitch to physical nic( tagged)
Payload (IP Packet)
06:00:01:AA:BB:CC
06:02:12:1D:1E
0x8100
0xA
0x800
46-1500 octets
Dest, addr
Src, addr
802.1Q
Tag
Type
*not all fields shown for clarity!

31. Ingress Traffic to VM!
From physical nic to vswitch( tagged)
Payload (IP Packet)
06:02:12:1D:1E:1F
06:00:01:AA:BB:CC
0x8100
0xA
0x800
46-1500 octets
From vswitch to VM A1 (untagged)
Payload (IP Packet)
06:02:12:1D:1E:1F
06:00:01:AA:BB:CC
0x800
46-1500 octets

32. VLAN networking!
Trunks! Trunks! Trunks!
User
A
User
A
User
A
User
User
B
A
User
B
…

33. 12 bits tag
=
4094 virtual networks

34. VLANs – other problems!
• Configuration complexity!
– Need to program switches carefully
• Large L2 domains!
– Broadcast in one VLAN can cause
unintended load on unrelated hypervisors
• Live migration limited to a single VLAN!
• Limited mac table sizes in L2 switches!
– 100s of vms per hypervisor
– 1000s of mac addresses on uplink port

35. Tunnels!
• Map VM address (Tenant Address) to Physical
address (PA) of Hypervisor!
– Software IPv4 tunnels between hypervisors
– Tunnel endpoints are PA of hypervisor
– Discriminator in tunnel header identifies tenant/
network
• GRE key in (NV) GRE tunnels (24-32 bits)
• VxLAN Network Identifier (VNI) in VxLAN (24 bits)
• Context ID in STT (64 bits)

36. GRE tunnel example!
GRE
Key
1
GRE
Key
2
OVS
User
1
OVS
User
1
User
1
OVS
User
OVS
User
OVS
2
1
User
2
…
…
…

37. GRE Example!
Hypervisor 1! Hypervisor 2!
VM
VM
VM
VM
A1
B1
A2
B2
192.168.10.55! 192.168.20.88! 192.168.10.5! 192.168.20.8!
vswitch
vswitch
10.10.10.5! 10.20.20.9!
10.20.20.9! 10.10.10.5! GRE key=10! MAC A2! MAC A1! 192.168.10.5! 192.168.10.55!
A1->A2!
B1->B2! 10.20.20.9! 10.10.10.5! GRE key=20! MAC B2! MAC B1! 192.168.20.8! 192.168.20.88!
Physical Address! Tenant L2 header! Tenant L3 header!
Wire format!

38. Layer 3 cloud networking!
Web
DB
Web
VM
VM
VM
Web
DB
Security
Security
Group
Group
Web
Web
DB
VM
VM
VM
…
…
…
Web
Web
VM
VM

39. L3 isolation with distributed firewalls!
!
Tenant 10.1.0.2
Public Public IP
1 VM 1
Internet address
65.37.141.11!
65.37.141.24! 10.1.0.1
!
Pod 1 Tenant 10.1.0.3
65.37.141.36!
!
Leaf 2 VM 1
65.37.141.80! Switch
!
!
Tenant 10.1.0.4
1 VM 2
L3 Core ! Pod 2
10.1.8.1
…!
!
Leaf
Switch
10.1.16.1
!
Load Pod 3
!
Balancer Leaf
Switch
… !

40. Networking in the Cloud Age!
SERVICES!

41. Virtual Network Services!
• Provide L2-L7 network services that
applications expect:!
– Load balancing, firewall, IDS, VPN, NAT, etc.
• Services are inserted in the virtual network
topology!
– usually in the path to the public network
• Services are on-demand (api-driven), scalable,
elastic!

42. Virtual Network Appliances!
Network services are often provided by virtual appliances.!
These are either commercial appliances in the virtual form factor or
Linux-based networking appliances!
Virtual Router!
Public Network Nic! Virtual Network Nic!
Control Network Nic!

43. Service insertion example!
Tenant 1 Virtual Network 10.1.1.0/24
Public Public IP
!
Tenant 10.1.1.2
Network address Gateway 1 VM 1
65.37.141.11! address 10.1.1.1
65.37.141.36
!
Tenant 1 ! Tenant 10.1.1.3
Edge 1 VM 2
Services
!
Appliance(s)
NAT!
Internet!
!
Tenant 10.1.1.4
DHCP!
1 VM 3
FW
!
Tenant 10.1.1.5
1 VM 4

44. Service insertion with VLAN !
Trunks! Trunks! Trunks!
Tena
nt
1
Tena
nt
1
Tena
nt
1
Tena
Tena
nt
2
Rout nt
1
er
VM
1
…
Public VLAN!
Public VLAN!
Public VLAN!
Rout
er
VM
2

45. Network Services!
Network
Services!
• L2
connectivity!
• IPAM!
• DNS!
• Routing!
• ACL!
• Firewall!
• NAT!
• VPN!
• LB!
• IDS!
• IPS!
!

46. Network Services!
Network Service
Services! Providers!
• L2 ü Virtual
connectivity! appliances!
• IPAM! ü Hardware
• DNS! firewalls!
• Routing! ü LB
• ACL! appliances!
• Firewall! ü SDN
• NAT! controllers!
• VPN! ü IDS /IPS
• LB! appliances!
• IDS! ü VRF!
• IPS! ü Hypervisor!
!

47. Network Services!
Network Service Network
Services! Providers! Isolation!
• L2 ü Virtual • No
connectivity! appliances!
• IPAM! isolation!
ü Hardware
• DNS! firewalls! • VLAN
• Routing! ü LB isolation!
• ACL! appliances!
• Firewall! ü SDN • Overlays!
• NAT! controllers! • L3
• VPN! ü IDS /IPS isolation!
• LB! appliances!
• IDS! ü VRF!
• IPS! ü Hypervisor!
!

48. Service Catalog!
• Cloud users are not exposed to the nature of
the service provider!
• Cloud operator designs a service catalog and
offers them to end users.!
– Gold = {LB + FW, using virtual appliances}
– Platinum = {LB + FW + VPN, using hardware
appliances}
– Silver = {FW using virtual appliances, 10Mbps}

49. End-user experience!
• Deploy a VM in a network!
– VM Template = Windows 2008 with Joomla
on VMWare!
– Service offering {m1.large} = 2 x CPU x
2.0Ghz, 8 GB RAM!
– Disk Offering {Super fast}!
– Network Offering {Gold} = Source NAT + LB+
FW + 20 Mbps Internet access!

50. End-user experience!
• Deploy a VM in a network!
– VM Template = Windows 2008 with Joomla on VMWare
– Service offering {m1.large} = 2 x CPU x 2.0Ghz, 8 GB
RAM
– Disk Offering {Super fast}
– Network Offering {Gold} = Source NAT + LB+ FW + 20
Mbps Internet access
• Network Offering Gold is realized by!
– VLAN isolation
– Source NAT & FW on Juniper SRX
– LB on F5 BigIp
– DHCP, DNS on virtual appliance

51. End-user experience!
• CloudStack orchestration:!
– Pick a free VLAN, pick a free public IP, free private IP
– Pick hypervisor with spare capacity
– Pick primary storage of SSD type accessible in hypervisor
cluster
– Pick a Juniper SRX and F5 with spare capacity
– Spin up a new virtual appliance if necessary that runs
DHCP and DNS service
• Pick hypervisor, call hypervisor APIs to provision virtual
appliance on selected VLAN
– Call hypervisor APIs to provision VM on selected VLAN
– Call SRX and F5 APIs to place their internal interfaces on
the VLAN, public interfaces on public VLAN
– Call SRX API to provision source NAT, default FW rules

52. Network services with VLANs!
Tenant 1 Virtual Network 10.1.1.0/24
!
Tenant 10.1.1.2
Gateway 1 VM 1
address 10.1.1.1
!
Tenant 10.1.1.3
1 VM 2
Internet!
!
Tenant 10.1.1.4
1 VM 3
!
Tenant 10.1.1.5
1 VM 4

53. Network virtualization with VLANs!
Tenant 1 Virtual Network 10.1.1.0/24
Public Public IP
!
Tenant 10.1.1.2
Network address Gateway 1 VM 1
65.37.141.11! address 10.1.1.1
65.37.141.36
!
Tenant 1 ! Tenant 10.1.1.3
Edge 1 VM 2
Services
Interne !
Appliance(s)
NAT!
!
Tenant
t! DHCP!
1 VM 3
10.1.1.4
FW
!
Tenant 10.1.1.5
1 VM 4

54. Network virtualization with VLANs!
Tenant 1 Virtual Network 10.1.1.0/24
Public Public IP
!
Tenant 10.1.1.2
Network address Gateway 1 VM 1
65.37.141.11! address 10.1.1.1
65.37.141.36
!
Tenant 1 ! Tenant 10.1.1.3
Edge 1 !
Tenant 1 VM 2
Edge
Services
Services
Appliance(s)
NAT!
! !
Appliance(s)
Internet!
!
Tenant 10.1.1.4
DHCP!
1 VM 3
FW
Load
Balancing!
!
VPN Tenant 10.1.1.5
1 VM 4

55. Service insertion with VLANs!
Tenant 1 Virtual Network 10.1.1.0/24
Public Public IP
!
Tenant 10.1.1.2
Network address Gateway 1 VM 1
65.37.141.11! address 10.1.1.1
65.37.141.36
!
Tenant 1 ! Tenant 10.1.1.3
Edge 1 !
Tenant 1 VM 2
Edge
Services
Services
Appliance(s)
NAT!
! !
Internet! Appliance(s)
!
Tenant 10.1.1.4
DHCP!
1 VM 3
FW
Load
Balancing!
!
Tenant 10.1.1.5
1 VM 4
Tenant 2 Virtual Network 10.1.1.0/24
Public IP
address
65.37.141.24!
Gateway
address
Tenant
2 VM 1 ! 10.1.1.2
65.37.141.80 10.1.1.1
!
Tenant 2 ! Tenant 10.1.1.3
Edge 2 VM 2
!
Services
Appliance
VPN!
NAT!
DHCP
Tenant
2 VM 3 ! 10.1.1.4

56. Scaling services with VLANs!
Scale out edge services using virtual appliances!
10.1.1.0/24!
VLAN 100
VM 1!
10.1.1.
2
65.37.141.1 10.1.1.1
11! CS!
65.37.141.1 Virtual VM 2!
12 Router! 10.1.1.
3
DHCP, DNS!
NAT!
Load 10.1.1.4 VM 3!
Balancing!
VPN
VM 4!
10.1.1.5

57. Scaling services with VLANs!
Scale out edge services using virtual appliances! Scale up using hardware devices!
10.1.1.0/24! 10.1.1.0/24!
VLAN 100 VLAN 100
VM 1! 65.37.141.11 10.1.1.1 10.1.1.2 VM 1!
10.1.1.
1 Juniper
2 SRX!
65.37.141.1 10.1.1.1
11! CS! Firewall! NAT,
65.37.141.1 Virtual VM 2! VPN! VM 2!
10.1.1. 10.1.1.3
12 Router!
3 65.37.141.11 10.1.1.112
DHCP, DNS! 2 Netscaler!
NAT! Load
Load 10.1.1.4 VM 3! VM 3!
Balancer! 10.1.1.4
Balancing!
VPN
VM 4! VM 4!
10.1.1.5 10.1.1.
5
CS!
DHCP, Virtual
Router!
DNS!

58. Multi-tier virtual networking!
Internet!
!
Loadbalancer Virtual appliance/!
Hardware Devices!
(virtual or HW)!
Network Services!
• IPAM!
• DNS! Web VM
1!
• LB [intra]!
• S-2-S VPN!
• Static Routes! Web VM
• ACLs! 2!
• NAT, PF!
• FW [ingress & egress]!
Web VM
3!
Web VM
4!
Web subnet !
10.1.1.0/24! VLAN 101

59. Multi-tier virtual networking!
Internet!
!
Loadbalancer Virtual appliance/!
Hardware Devices!
(virtual or HW)!
Network Services!
App VM
• IPAM!
1!
• DNS! Web VM
1!
• LB [intra]!
• S-2-S VPN! App VM
• Static Routes! Web VM 2! VLAN 2724
• ACLs! 2!
• NAT, PF!
• FW [ingress & egress]! VLAN 353
Web VM DB VM
• BGP! 3! 1!
Web VM
4!
Web subnet ! App subnet DB Subnet!
10.1.1.0/24! VLAN 101
10.1.2.0/24! 10.1.3.0/24!

60. Multi-tier virtual networking!
Internet!
IPSec or SSL site-to-site VPN!
! Custome
Loadbalancer Virtual appliance/!
r!
Hardware Devices!
(virtual or HW)! Premises!
MPLS VLAN!
Network Services!
App VM
• IPAM!
1!
• DNS! Web VM
1!
• LB [intra]!
• S-2-S VPN! App VM
• Static Routes! Web VM 2! VLAN 2724
• ACLs! 2!
• NAT, PF!
• FW [ingress & egress]! VLAN 353
Web VM DB VM
• BGP! 3! 1!
Web VM
4!
Web subnet ! App subnet DB Subnet!
10.1.1.0/24! VLAN 101
10.1.2.0/24! 10.1.3.0/24!

61. Multi-tier networking with
Overlay!
Internet!
IPSec or SSL site-to-site VPN!
Loadbalancer ! Custome
(virtual Virtual Router! r!
Premises!
appliance)!
MPLS VLAN!
Network Services! App VM
• IPAM! Web VM
1!
• DNS! 1!
• LB [intra]!
App VM
• S-2-S VPN! 2!
• Static Routes! Web VM GRE Key 2724
2!
• ACLs!
• NAT, PF!
• FW [ingress & egress]! Web VM GRE Key 353
DB VM
• BGP! 3! 1!
Web VM
4!
Web subnet ! App subnet DB Subnet!
10.1.1.0/24! GRE Key 101
10.1.2.0/24! 10.1.3.0/24!

62. Multi-tier networking with
Overlay!
Internet!
Loadbalancer vswitches!
(virtual
appliance)!
App VM
1!
Web VM
1!
App VM
Web VM 2! GRE Key 2724
2!
Web VM GRE Key 353
DB VM
3! 1!
Web VM
4!
Web subnet ! App subnet DB Subnet!
10.1.1.0/24! GRE Key 101
10.1.2.0/24! 10.1.3.0/24!

63. Networking in the Cloud Age!
LAYER 3 ISOLATION!

64. Layer 3 cloud networking!
Web DB Web
VM! VM! VM!
Web! DB !
Security Security
Group! Group!
Web Web DB
VM! VM! VM!
…! …! …!
Web Web
VM! VM!
Ingress Rule: Allow VMs in Web Security Group access to VMs in DB Security Group on Port 33

65. L3 isolation with distributed firewalls!
!
Tenant 10.1.0.2
Public Public IP
1 VM 1
Internet address
65.37.141.11!
65.37.141.24! 10.1.0.1
!
Pod 1 Tenant 10.1.0.3
65.37.141.36!
!
Leaf 2 VM 1
65.37.141.80! Switch
!
!
Tenant 10.1.0.4
1 VM 2
L3 Core ! Pod 2
10.1.8.1
…!
!
Leaf
Switch
10.1.16.1
!
Load Pod 3
!
Balancer Leaf
Switch
… !

66. L3 isolation with distributed firewalls!
!
Tenant 10.1.0.2
Public Public IP
1 VM 1
Internet address
65.37.141.11!
65.37.141.24! 10.1.0.1
!
Pod 1 Tenant 10.1.0.3
65.37.141.36!
!
Leaf 2 VM 1
65.37.141.80! Switch
!
!
Tenant 10.1.0.4
1 VM 2
L3 Core ! Pod 2
10.1.8.1
…!
!
Leaf
Switch
10.1.16.1
!
Load Pod 3
!
Balancer Leaf
Switch
… !
Tenant
1 VM 3 ! 10.1.16.47
!
Tenant
10.1.16.85
1 VM 4

67. L3 isolation with distributed firewalls!
!
Tenant 10.1.0.2
Public Public IP
1 VM 1
Internet address
65.37.141.11!
65.37.141.24! 10.1.0.1
!
Pod 1 Tenant 10.1.0.3
65.37.141.36!
!
Leaf 2 VM 1
65.37.141.80! Switch
!
!
Tenant 10.1.0.4
1 VM 2
L3 Core ! Pod 2
10.1.8.1
…!
!
Leaf
Switch
!
Tenant 10.1.16.12
10.1.16.1 2 VM 2
!
Load Pod 3
!
Balancer Leaf
!
Switch Tenant
2 VM 3 10.1.16.21
… !
Tenant
1 VM 3 ! 10.1.16.47
!
Tenant
10.1.16.85
1 VM 4

68. 1 Firewall per Virtual Machine

69. A Million Firewalls?!
VM! VM! VM!
…! …! VM! VM!
…! …!
VM! VM! …!
VM! VM!
VM! VM!
VM! VM! VM! VM!
VM! VM! VM!
…! …! VM! VM!
…! …!
VM! VM! …!
VM! VM!
VM! VM!
VM! VM! VM! VM!
VM! VM! VM!
…! …! VM! VM!
…! …!
VM! VM! …!
VM! VM!
VM! VM!
VM! VM! VM! VM!
VM! VM! VM!
…! …! VM! VM!
…! …!
VM! VM! …!
VM! VM!
VM! VM!
VM! VM! VM! VM!
VM! VM! VM!
…! …! VM! VM!
…! …!
VM! VM! …!
VM! VM!
VM! VM!
VM! VM! VM! VM!
VM!
…!
VM!
VM!
…!
VM!
VM!
…!
VM!
VM!
…! …! VM!
…!
VM! VM!
VM! VM! VM! VM! VM!
VM! VM! VM!
…! …! VM! VM!
…! …!
VM! VM! …!
VM! VM!
VM! VM!
VM! VM! VM! VM!
VM! VM! VM!
…! …! VM! VM!
…! …!
VM! VM! …!
VM! VM!
VM! VM!
VM! VM! VM! VM!

70. Networking in the Cloud Age!
SOFTWARE DEFINED
NETWORKING!

71. Definition!
• Separation of Control Plane from the hardware
performing the forwarding function!
• Control plane is logically centralized!

72. SDN Advantages!
• Centralized control makes it easier to
configure, troubleshoot and maintain
• Eliminates ‘box’ mode of configuration
• Enables control at a high level

73. Related to SDN!
• API layer over a collection of ‘boxes’!
– API layer communicates with boxes using box-level
APIs / ssh / telnet
• OpenFlow!
– Standard protocol for the centralized control plane to
talk to the forwarding elements.
• Tunnels / overlays!
– SDN is valuable for virtual topologies
– Initial target of SDN implementation

74. Centralized control plane!
Admin/User
API
Controller
Cluster
MySQL/NoSQL
Openflow/ssh/netconf/other!
Boxes!

75. SDN problems!
• Discovery of virtual address -> physical
address mapping!
– VxLAN = multicast
– GRE = programmed by control plane
– L3 isolation = no mapping, no discovery

76. SDN problems!
• State maintenance!
– Large number of endpoints + flows
– High arrival rate of new flows
– Needs fast and scalable storage and
processing

77. CloudStack and SDN!
Hypervisor
Hypervisor
Resource
5
4
Resource
Hyperviso
Hyperviso
r
Plugins
r
Plugins
Plugin
Framew 6
ork
Network
API
7
SDN
Resource
Network
API
Network
controller
OrchestraSon
Engine
Plugins
1
API
Plugins
2
8
Allocator
9
3
Storage
Plugins
Plugins
Storage
Storage
Resource
Resource
Allocator
Allocator
Plugins
Plugins
Physical Resources !
Network plugin is the glue that understands the SDN controller’s API!

78. Virtual Networking "!
NETWORKING IN APACHE
CLOUDSTACK!

79. Regions and Zones!
• A cluster of CloudStack management servers
manage the physical resources of a region
– Single API endpoint per region
• Each region consists of zones
• Zones are physically proximate, but provide
distinct failure domains (e.g., flood, earthquake,
power)
• Zones are interconnected with high speed low
latency links

80. Region “West”
Region “East”
Geographic
separation
Internet
Low Latency
Region “South”

81. Region “West”
Zone “West-Beta”
Zone “West-Alpha”
High Speed Backbone
(e.g., SONET ring)
Zone “West-Delta”
Zone “West-Gamma”

82. Inside a zone!
Admin/User
API
End
users
CloudStack
Cluster
DC
Edge
MySQL
L2/L3
core
Leaf
Sw
Hypervisor
(Xen
/VMWare/KVM)
Secondary
Storage
Primary
Storage
NFS/ISCSI/FC
Pod
Pod
Pod
Pod
Pod

83. Orchestration!
• Orchestration describes the automated
arrangement, coordination, and management of
complex computer systems, middleware and
services
– Wikipedia!

84. CloudStack Architecture!
Hypervisor
Hypervisor
Plugins
Plugins
Plugin
Framework
Network
Plugins
OrchestraSon
Engine
Network
Plugins
Allocator
Allocator
Plugins
Plugins
Storage
Plugins

85. CloudStack Architecture!
• XenServer
• VMWare
• KVM
• OracleVM
Hypervisor
Hypervisor
Plugins
Plugins
Plugin
Framework
Nicira
•
• Netscaler
• Brocade
Network
Plugins
OrchestraSon
Engine
Network
Plugins
idoNet
• M
Allocator
• Random
Allocator
• User-­‐
Plugins
Plugins
concentrated
• Intel
TXT
• Affinity

86. CloudStack Architecture!
Hypervisor
Hypervisor
Resource
5
4
Resource
Hyperviso
Hyperviso
r
Plugins
r
Plugins
Plugin
Framew 6
ork
Network
API
7
Network
Resource
Network
API
Network
Resource
OrchestraSon
Engine
Plugins
1
API
Plugins
2
8
Allocator
9
3
Storage
Plugins
Plugins
Storage
Storage
Resource
Resource
Allocator
Allocator
Plugins
Plugins
Physical Resources !
Orchestration steps can be executed in parallel or in sequence!

87. Problem:
Manage Configuration of!
1000s of virtual appliances (or VRF)
Dozens of HW appliances
Solution:
Database-driven state management of
appliances!
Message queues + Retry Logic
Idempotent updates,
Recreatable virtual appliances

88. Problem:
Manage Configuration of!
1000s of virtual appliances (or VRF)
Dozens of HW appliances
Solution:
Database-driven state management of
appliances!
Message queues + Retry Logic!
Idempotent updates,!
Recreatable virtual appliances!
!

89. Problem:!
Single-tenant HW appliances!
Solution:!
CloudStack API layers multi-tenancy, provides
abstraction!
No direct access to devices!

90. Problem:
Hardware appliances with no APIs
CLI only
Limited concurrent login sessions
Solution:
Recommend appliances with APIs
Integrate with Network Orchestrators
!

91. Problem:
Manage the configuration of 100s of thousands of firewalls
Solution:
Well-known software scaling techniques
• Message queues
• Consistency tradeoffs
• Idempotent configuration & retries
CloudStack uses
• special purpose queues
• optimized for large security groups
• eventual consistency for rule updates

92. Problem:
Firewall (iptables) rules explosion on the host firewall!
Allow Security Group {Web} on TCP port 3060 !
!
-A FORWARD -m tcp –p tcp –dport 3060 –src 10.1.16.31 – j ACCEPT
-A FORWARD -m tcp –p tcp –dport 3060 –src 10.1.45.112 – j ACCEPT
-A FORWARD -m tcp –p tcp –dport 3060 –src 10.1.189.5 – j ACCEPT
…!
-A FORWARD -m tcp –p tcp –dport 3060 –src 10.21.9.77 – j ACCEPT
For large security groups, performance suffers

93. Problem:
Firewall (iptables) rules explosion on the host firewall
!
Solution:!
Use ipsets: !
ipset –N web_sg iptreemap
ipset –A web_sg 10.1.16.31
ipset –A web_sg 10.1.16.112
ipset –A
…!
web_sg 10.1.189.5
ipset –A web_sg 10.21.9.77
-A FORWARD –p tcp –m tcp –dport 3060 –m set –match-set web_sg src -j ACCEPT

94. Apache CloudStack!
• Apache CloudStack!
– http://www.cloudstack.org/!
– Download it!
– Use it!
– Contribute to it!
• Citrix CloudPlatform!
– Based on Apache CloudStack!
– Commercial support!