Evolution of CloudStack Architecture (Collab 2012)

The Evolution of
CloudStack
The roads taken and not taken
@chiradeep
Collab 2012

Saturday, December 1, 2012 1

We remember the most painful and pleasurable
parts of an experience, not the duration


In the Beginning


Circa
2008


Circa
2008

!"#$%&'(


Circa
2008

!"#$%&'( !"#$%&'(


Circa
2008

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
2008

Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
2008

Open Source Xen
Security Groups
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
2008

Open Source Xen
(custom)
Security Groups EBS

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
2008

Open Source Xen
(custom)
Security Groups EBS S3

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
2008

Proprietary Orchestration Services

Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
2008

EC2
API


Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
2008

EC2 EBS
API API


Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
2008

EC2 EBS
S3 API
API API


Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
2008

EC2 EBS
S3 API
API API


Secret
Sauce Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


The
Virtual
Datacenter
OS
allows
businesses
to

eﬃciently
pool
all
types
of
hardware
resources
-‐

servers,
storage
and
network
–
into
an
aggregated

on-‐premise
cloud
-‐
VMWare
Press
Release
9/2008


The
Virtual
Datacenter
OS
allows
businesses
to

eﬃciently
pool
all
types
of
hardware
resources
-‐

servers,
storage
and
network
–
into
an
aggregated

on-‐premise
cloud
-‐
VMWare
Press
Release
9/2008

Eucalyptus
is
the
only
cloud
architecture
to
support

the
same
applicaKon
programming
interfaces
(APIs)

as
public
clouds,
and
today
Eucalyptus
is
fully

compaKble
with
the
Amazon
AWS
public
cloud

infrastructure.

Eucalyptus
Systems
Press
Release
4/2009


2.0 AWS 2.2 3.0 Quality 4.X
Prototype 1.0 GA
Refactor Compatibility Refactor Improvements Refactor

2008 2009 2010 2011 2012

Sept 2008: Nov 2009: May 2010: July 2011: April 2012:
VMOps CloudStack Cloud.com Citrix Apache
Founded 1.0 GA Launch & Acquires CloudStack
CloudStack Cloud.com
2.0 GA


Circa
3/2009


Circa
3/2009

!"#$%&'(


Circa
3/2009

!"#$%&'( !"#$%&'(


Circa
3/2009

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

Open Source Xen
Virtual Networks
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

Open Source Xen
(custom)
Virtual Networks NFS

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

Proprietary Control Plane

Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009



Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

GUI



Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

GWT
GUI



Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

GWT
GUI

Java App
Server


Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

GWT
GUI

Java App
Server

JSON
over TCP

Open Source Xen
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

GWT
GUI

Java App
Server

JSON
over TCP

NFS (ZFS on
Open Source Xen OpenSolaris)
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

GWT
GUI

Java App
Server

JSON
over TCP
UDP
Tunnels NFS (ZFS on
(custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

GWT
GUI

Java App
Server

JSON
over TCP
UDP
Tunnels NFS (ZFS on
Agent (custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2009

GWT
GUI

Java App
Server

JSON
over TCP
UDP
Tunnels NFS (ZFS on
Open Source Xen Agent OpenSolaris)
Agent (custom)

!"#$%&'( !"#$%&'( !"#$%&'(


Availability Zone Architecture



L3 core


End users

DC Edge

L3 core


End users

DC Edge

L3 core

Access Sw

Hypervisor (Xen)

Primary Storage
ZFS

Pod


End users

DC Edge

L3 core

Access Sw

Hypervisor (Xen)

Primary Storage
ZFS

Pod Pod


End users

DC Edge

L3 core

Access Sw

Hypervisor (Xen)

Primary Storage
ZFS

Pod Pod Pod


End users

DC Edge

L3 core

Access Sw

Hypervisor (Xen)

Primary Storage
ZFS

Pod Pod Pod Pod


End users

DC Edge

L3 core

Access Sw

Hypervisor (Xen)

Primary Storage
ZFS

Pod Pod Pod Pod Pod


End users
Admin/User API

CloudStack
DC Edge
MySQL

L3 core

Access Sw

Hypervisor (Xen)

Primary Storage
ZFS

Pod Pod Pod Pod Pod


End users
Admin/User API

CloudStack
DC Edge
MySQL
ECMP/BGP
L3 core

Access Sw

Hypervisor (Xen)

Primary Storage
ZFS

Pod Pod Pod Pod Pod


Multiple zones
CloudStack

MySQL

Zone1


Multiple zones
CloudStack

MySQL

Internet

Zone1 Zone2 Zone3


Network Virtualization


Tenant

1
Virtual
Network
10.1.1.0/24

Tenant
1

10.1.1.2
Gateway
address
VM
1
10.1.1.1

Tenant
1

10.1.1.3
VM
2

Tenant
1

10.1.1.4
VM
3

Tenant
1

10.1.1.5
VM
4


Tenant

1
Virtual
Network
10.1.1.0/24

Tenant
1

10.1.1.2
Gateway
address
VM
1
10.1.1.1

Tenant
1
Tenant
1

10.1.1.3
Virtual
Router VM
2

NAT
Internet DHCP Tenant
1

10.1.1.4
FW VM
3
Load
Balancing

Tenant
1

10.1.1.5
VM
4


Tenant

1
Virtual
Network
10.1.1.0/24

Public
IP
address
Tenant
1

10.1.1.2
65.37.141.11 Gateway
address
VM
1
65.37.141.36 10.1.1.1
Public
Network
Tenant
1
Tenant
1

10.1.1.3
Virtual
Router VM
2

NAT
1

10.1.1.4
FW VM
3
Load
Balancing

Tenant
1

10.1.1.5
VM
4


Tenant

1
Virtual
Network
10.1.1.0/24

Public
IP
address
Tenant
1

10.1.1.2
65.37.141.11 Gateway
address
VM
1
65.37.141.36 10.1.1.1
Public
Network
Tenant
1
Tenant
1

10.1.1.3
Virtual
Router VM
2

NAT
1

10.1.1.4
FW VM
3
Load
Balancing

Tenant
1

10.1.1.5
VM
4

Tenant
2
Virtual
Network
10.1.1.0/24
Public
IP
address

65.37.141.24 Gateway
address
Tenant
2

10.1.1.2
10.1.1.1 VM
1
65.37.141.80

Tenant
2
Tenant
2

10.1.1.3
Virtual
Router
VM
2
Appliance

FW
NAT Tenant
2

10.1.1.4
DHCP VM
3


Virtual Machine Placement
End users

CloudStack
DC Edge
MySQL

L3/L2 core

Hypervisor (Xen)

Primary Storage
ZFS

Pod Pod Pod Pod Pod


Network Virtualization - why?

AWS-style security groups “unfamiliar”

Adopt traditional L2 model for end-users

VLANs do not scale

use network virtualization to realize this


UDP encapsulation of Ethernet Frames

similar to VXLAN (128-bit address vs 24-bit
for VXLAN)

Driver in dom0

Virtual Router to provide edge services

including Load Balancing

Hand-rolled Fedora 8 para-virtualized


ZFS (Volume Service)

ZFS for highly scalable, reliable storage

RAID-Z

SSD cache

NFS for shared storage

Hand-rolled version of OpenSolaris 2008.11


Orchestration


Orchestration
JVM


Orchestration
JVM

MySQL


Orchestration
JVM

Message Bus

MySQL


Orchestration
JVM
Front-
end

Message Bus

MySQL


Orchestration
JVM
VM
Front-
Orchest
end
rator

Message Bus

MySQL


Orchestration
JVM
VM Network
Front-
Orchest Orchestr
end
rator ator

Message Bus

MySQL


Orchestration
JVM
VM Network Storage
Front-
Orchest Orchestr Orchest
end
rator ator rator

Message Bus

MySQL


Orchestration
JVM
VM Network Storage Host/
Front-
Orchest Orchestr Orchest Resource
end
rator ator rator Manager

Message Bus

MySQL


Orchestration
JVM
Front-
end

Message Bus

Hypervi
Hypervi
Hypervisor
sor
Agent
sor
(JVM)

MySQL


Orchestration
JVM
Front-
end

Message Bus

Hypervi
Hypervi
Hypervisor Storage
Storage
Storage
sor
Agent
sor Agent
Agent
Agent
(JVM) (JVM)

MySQL


Orchestration
JVM
Front-
end

Message Bus

Hypervi
Hypervi
Hypervisor Storage
Storage
Storage Console
Console
sor
Agent
sor Agent
Agent
Agent Agent
Agent
(JVM) (JVM)

MySQL


Orchestration
JVM
GWT-RPC Front-
Browser Orchest Orchestr Orchest Resource
end

Message Bus

Hypervi
Hypervi
Hypervisor Storage
Storage
Storage Console
Console
sor
Agent
sor Agent
Agent
Agent Agent
Agent
(JVM) (JVM)

MySQL


Reject Pile (2009)
Local disks of hypervisors presented as
network block storage with network RAID

Too complex (sheepdog-like)

Split-brain issues

DRBD

Too complex to automate recovery

Commercial Arrays

Limited volumes, limited snapshots

Reject Pile (2009)
Security Groups

Not well understood

VLANs

4K limit

Physical Network Appliances

No APIs (Cisco at least)

Can’t decapsulate UDP tunnel


Reject Pile (early 2009)

KVM

Unstable

Windows support

No CPU allocation


April 2009


April 2009
“5 guys in
a garage”


Ship It! ( Beta 6/2009)

ISO/CD with Xen 3.3 / CentOs customized

ISO/CD with OSOL 2008.11 + patches +
drivers

Java binaries

System VM: Fedora Core 8 VM bits including
dnsmasq, iptables, HAProxy

Installers, documentation


Road to 1.0
ISCSI - ZFS zvols -> ∞ snapshots

Performance tuning (UDP tunnels, zvol)

Usage, metering,

Formal user / admin Query-based API

JQuery-based GUI - skinnable

Windows PV drivers


More rejects (2009)

Network RAID of zvols over ISCSI/md

Performance

Re-mirror time / penalty

Consistent snapshots

Split-brain


1.0 - Ship It (Nov 2009)

6 engineers + 3 QA + 1 sales

3 beta sites

Traction with SPs interested in competing
with AWS


Orchestration + Usage server

JVM

Front-
end



JVM MySQL

Front-
end


Browser

LoadBalan
JVM MySQL

ced
HTTP
Query Front-
API end


Browser

LoadBalan
JVM MySQL

ced
HTTP VM Network Storage Host/
Query Front-
API
end


Browser

LoadBalan
JVM MySQL

ced
HTTP VM Network Storage Host/
Query Front-
API
end

Message Bus

Hypervi
Hypervi
Hypervisor Storage
Storage
Storage Console
Console
sor
Agent
sor Agent
Agent
Agent Agent
Agent
(JVM) (JVM)


Browser

LoadBalan
JVM MySQL

ced Storage
VM
VM Network
Network Storage
HTTP Front-
Front- Orchest Network Storage Host/
VM Orchestr Orchest Host/
Query Front- Orchest Orchestr Orchest Resource
end
API end Orchest Orchestr rator
rator ator Orchest Resource
end rator ator rator
Manager

Clustered
Message Bus
Orchestration Message Bus
Message Bus
Server

Hypervi
Hypervi
Hypervisor Storage
Storage
Storage Console
Console
sor
Agent
sor Agent
Agent
Agent Agent
Agent
(JVM) (JVM)


Browser

LoadBalan
JVM MySQL

ced Storage
VM
VM Network
Network Storage
HTTP Front-
Front- Orchest Network Storage Host/
VM Orchestr Orchest Host/
Query Front- Orchest Orchestr Orchest Resource
end
API end Orchest Orchestr rator
rator ator Orchest Resource
end rator ator rator
Manager

Usage
Clustered
Message Bus
Orchestration Message Bus
Message Bus
Server

Hypervi
Hypervi
Hypervisor Storage
Storage
Storage Console
Console
sor
Agent
sor Agent
Agent
Agent Agent
Agent
(JVM) (JVM)


1.0 Issues
Multicast scaling

Oracle acquisition of OpenSolaris

Hiring “full-stack” developers

5 startups - Hypervisor + Storage + Network
Virtualization + Network appliance +
Orchestration


Multicast Scaling
Broadcast (ARP) / Multicast from VM
translates to UDP multicast

Switches snoop on IGMP to prevent ﬂooding
all ports

Most switches fall-over after 64 Multicast
groups

each tenant -> 1 multicast group


Road Ahead
Multi-hypervisor, SAN, and VLAN support

Use commercial XenServer

Commercial ﬁlers / ISCSI arrays

Support KVM + NFS

Product ﬁrst, architecture second

From web hosting to enterprise workload


Road Ahead

Throw out custom hypervisor and
OpenSolaris

VLANs

Secondary Storage to store snapshots and
images


Circa
3/2010

GUI



Open Source Xen Storage
Agent
Virtual Networks
Agent (custom) Arrays

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2010

JQuery
GUI



Agent
Virtual Networks

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2010

JQuery
GUI



VLANs
Agent
Virtual Networks

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2010

JQuery
GUI



VLANs
Virtual Networks

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2010

JQuery
GUI



VLANs
Virtual Networks
(custom) Arrays

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2010

JQuery
GUI



VLANs
Storage
Virtual Networks
Arrays

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2010

JQuery
GUI



VLANs
Commercial Storage
Virtual Networks
XenServer Arrays

!"#$%&'( !"#$%&'( !"#$%&'(


Circa
3/2010

JQuery
GUI


ISCSI/
VLANs NFS
Commercial Storage
Virtual Networks
XenServer Arrays

!"#$%&'( !"#$%&'( !"#$%&'(


Browser

LoadBalan
JVM MySQL

ced Front-
HTTP end VM Network Storage Host/
Query
API
Job Queue
Usage
Clustered
Message Bus
Orchestration
Server

Hypervi
XS
sor
Agent

XAPI
KVM Console
Console Agent
Secondary
Agent Agent Storage Agent
XenServers (JVM)
(JVM)
(JVM)


End users
Admin/User API

CloudStack
DC Edge
MySQL

Access Sw

Hypervisor (Xen
/VMWare/KVM)

Pod Pod Pod Pod Pod


End users
Admin/User API

CloudStack
DC Edge
MySQL

Access Sw

Hypervisor (Xen
/VMWare/KVM)
Secondary Storage

Pod Pod Pod Pod Pod


End users
Admin/User API

CloudStack
DC Edge
MySQL
L2/L3 core

Access Sw

Hypervisor (Xen
/VMWare/KVM)
Secondary Storage

Pod Pod Pod Pod Pod


End users
Admin/User API

CloudStack
DC Edge
MySQL
L2/L3 core

Access Sw

Hypervisor (Xen
/VMWare/KVM)
Secondary Storage

Primary Storage
NFS/ISCSI/FC

Pod Pod Pod Pod Pod


System VM

Data Path problem

Need to move bits between storage types

Present consoles to end users

Scale up / down these data path services


System VMs

Virtual Routers, Console Proxy VMs,
Secondary Storage VMs

based on the same vm image

Orchestration and scaling baked in


2.0 6/2010

Shipping just java binaries + system vm
image

Open sourced (GPL)

Company name changed to Cloud.com

New product - Cloud Portal


Citrix Cloud Portal
Business and operations support platform for
Service Providers

Account management

Self-service portal

Billing and metering

CRM and support ticketing

Not OSS


Reject Pile (2010)
“Zone” and “Pod” VLANs to go beyond 4k
limit

Pod VLANs bridged using zone VLANs

Another SDN attempt using GRE on Open
vSwitch


Flexibility and Scale
(2010-2011)

Network, hypervisor and vm placement
ﬂexibility

Tens of thousands of hypervisors

AWS API


Network Flexibility (2.2
refactor)

Hardware appliances instead of virtual router

Upgrade / downgrade to/from hardware to
virtual

Pick and choose services (L2-L7)

Security Groups


Network Flexibility
Scale
out
edge
services
using
virtual
appliances

10.1.1.0/24
VLAN
100

VM
1
10.1.1.2

65.37.141.111 10.1.1.1
65.37.141.112
CS
Virtual
10.1.1.3
VM
2
Router
DHCP,
DNS
NAT
10.1.1.4 VM
3
Load
Balancing

VM
4
10.1.1.5


Network Flexibility
Scale
out
edge
services
using
virtual
appliances Scale
up
using
hardware
devices
10.1.1.0/24
10.1.1.0/24 VLAN
100
VLAN
100

65.37.141.111 10.1.1.1 10.1.1.2 VM
1
VM
1
10.1.1.2
Juniper
SRX
65.37.141.111 10.1.1.1 Firewall NAT,

65.37.141.112 VPN
CS
VM
2
Virtual
10.1.1.3
VM
2 10.1.1.3
Router 65.37.141.112 10.1.1.112
DHCP,
DNS Netscaler
NAT Load

10.1.1.4 VM
3 VM
3
Load
Balancing Balancer 10.1.1.4

VM
4 VM
4
10.1.1.5 10.1.1.5

CS
DHCP,

Virtual

DNS
Router


VM Placement

Customize vm placement

Pick hosts and storage according to rules

Afﬁnity / anti-afﬁnity


Multi-hypervisor

XenServer, KVM and VMWare in the same
zone

Differences in network and storage behavior

Different image formats

Same system vm image


Plugin Architecture (2.2)

Orchestration
Engine



Plugin
Framework

Orchestration
Engine


Hypervisor
Hypervisor
Plugins
Plugins
Plugin
Framework

Orchestration
Engine


Hypervisor
Hypervisor
Plugins
Plugins
Plugin
Framework

Orchestration Network
Network
Plugins
Plugins
Engine


Hypervisor
Hypervisor
Plugins
Plugins
Plugin
Framework

Network
Plugins
Plugins
Engine
Allocator
Allocator
Plugins
Plugins


Hypervisor
Hypervisor
Plugins
Plugins
Plugin
Framework

Network
Plugins
Plugins
Engine
Allocator
Allocator
Plugins
Plugins

Storage
Plugins


Plugin Architecture
Hypervisor
Hypervisor
Plugins
Plugins
Plugin
Framework

Network
Plugins
Plugins
Engine
Allocator
Allocator
Plugins
Plugins


Plugin Architecture •XenServer
•VMWare

•KVM

•OracleVM

Hypervisor
Hypervisor
Plugins
Plugins
Plugin
Framework

Network
Plugins
Plugins
Engine
Allocator
Allocator
Plugins
Plugins


•VMWare

•KVM

•OracleVM

Hypervisor
Hypervisor
Plugins
Plugins
Plugin •Nicira
Framework •Netscaler

Network
Plugins
•Brocade

•MidoNet
Plugins
Engine
Allocator
Allocator
Plugins
Plugins


•VMWare

•KVM

•OracleVM

Hypervisor
Hypervisor
Plugins
Plugins
Plugin •Nicira
Framework •Netscaler

Network
Plugins
•Brocade

•MidoNet
Plugins
Engine
Random
•
Allocator
Allocator •User-
Plugins
Plugins
concentrated
•Intel TXT

•Afﬁnity


Security Groups

Web
DB
Web

VM VM VM

Web
Web
DB

VM VM VM

… … …

Web
Web

VM VM

Ingress
Rule:
Allow
VMs
in
Web
Security
Group
access
to
VMs
in
DB
Security
Group
on
Port
3306


Security Groups

Web
DB
Web

VM VM VM
Web
Security

Group
Web
Web
DB

VM VM VM

… … …

Web
Web

VM VM

Ingress
Rule:
Allow
VMs
in
Web
Security
Group
access
to
VMs
in
DB
Security
Group
on
Port
3306


Security Groups

Web
DB
Web

VM VM VM
Web DB

Security
Security

Group
Group
Web
Web
DB

VM VM VM

… … …

Web
Web

VM VM

Ingress
Rule:
Allow
VMs
in
Web
Security
Group
access
to
VMs
in
DB
Security
Group
on
Port
3306


A million firewalls?

Manage the config state of 10^6 firewalls?

“Eventual consistency”

Iptable Rule explosion on hypervisor

use ‘ipset’ to optimize lookup


2.1 -> 3.02

Features, features features

VMWare, Oracle VM, Baremetal

Multi-tier networking, VPC, VPN, more
vendors

EC2 API, S3 API

SDN

Focus on quality

Evolution of CloudStack Architecture (Collab 2012)

Evolution of CloudStack Architecture (Collab 2012)

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (12)

More from Chiradeep Vittal

More from Chiradeep Vittal (11)

Recently uploaded

Recently uploaded (20)

Evolution of CloudStack Architecture (Collab 2012)