1. CloudStack Architecture
4/29/2012
Chiradeep Vittal
Alex Huang

2. Outline
• Overview of CloudStack
• Problem Definition
• Feature set overview
• System VMs
• System Architecture & Context
• Component View

3. What is CloudStack?
• Secure, multi-tenant cloud
orchestration platform
– Turnkey platform for delivering
IaaS clouds
Build your cloud the way the – Hypervisor agnostic
world’s most successful clouds
are built – Scalable, secure and open
– Open source, open standards
– Deploys on premise or as a hosted
solution
• Deliver cloud services faster
and cheaper

4. CloudStack Supports Multiple Cloud Strategies
Private Clouds Public Clouds
On-premise Hosted Multi-tenant
Enterprise Cloud Enterprise Cloud Public Cloud
• Dedicated • Dedicated • Mix of shared and
resources resources dedicated
• Security & total • Security resources
control • SLA bound • Elastic scaling
• Internal network • 3rd party owned • Pay as you go
• Managed by and operated • Public internet,
Enterprise or 3rd VPN access
party

5. CloudStack Provides On-demand Access to
Infrastructure Through a Self-Service Portal
Org A Org B
Users
Admin Admin
End User Users Users
Compute Network Storage
Admin

6. Open Flexible Platform
Compute Hypervisor
XenServer VMware Oracle VM KVM Bare metal
Storage Block & Object
Fiber
Local Disk iSCSI NFS Swift
Channel
Primary Storage Secondary Storage
Network Network & Network Services
Network Load
Isolation Firewall VPN
Type balancer

7. Problem Definition
• Offer a scalable, flexible, manageable IAAS platform that
follows established cloud computing paradigms
• IAAS
– Orchestrate physical and virtual resources to offer self-service
infrastructure provisioning and monitoring
• Scalable
– 1 -> N hypervisors / VMs / virtual resources
– 1 -> N end users
• Flexible
– Handle new physical resource types
• Hypervisors, storage, networking
– Add new APIs
– Add new services
– Add new network models

8. Problem Definition (contd)
• Manageable
– Hide complexity of underlying resources
– Rich functional end-user and admin UI
– Admin API to automate operations
– Easy install, upgrade for small -> large clouds
– Simple scaling, automated resilience
• Established Paradigms
– EC2 –inspired
• Semantic variations based on cloud provider
needs, hypervisor capabilities

9. End-User Experience
Self-service Portal to Create &
Manage VMs

10. Create Custom Virtual Machines via
Service Offerings Select Operating System
• Windows, Linux
Select Compute Offering
• CPU & RAM
Select Disk Offering
• Volume Size
Select Network Offering
• Network & Services
Create VM

11. Dashboard Provides Overview of Consumed
Resources
• Running, Stopped &
Total VMs
• Public IPs
• Private networks
• Latest Events

12. Virtual Machine Management
Users
Change
VM Operations Console Access VM Status
Service Offering
Start
• CPU Utilized 2 CPUs 4 CPUs
Stop 1 GB 4 GB
• Network Read RAM RAM
Restart • Network Writes 20 GB 200 GB
Destroy 20 Mbps 100
Mbps

13. Volume & Snapshot Management
VM 1
Add / Delete
Volumes Volume
Create Templates Volume Template
from Volumes
Hourly Weekly
Schedule Now
Snapshots Daily Monthly
….
View Snapshot
History

14. Network & Network Services
• Create Networks and attach VMs
• Acquire public IP address for NAT &
load balancing
• Control traffic to VM using ingress
and egress firewall rules
• Set up rules to load balance traffic
between VMs

15. CloudStack Deployment Architecture
CloudStac
Inter
k net  Hypervisor is the basic unit of
Manage scale.
ment
Zone 1 Server  Cluster consists of one ore
more hosts of same hypervisor
L3 core
 All hosts in cluster have access
to shared (primary) storage
Pod 1 Access Layer Pod N
Secondary
 Pod is one or more clusters,
…. Storage usually with L2 switches.
Cluster N
 Availability Zone has one or
more pods, has access to
…. secondary storage.
 One or more zones represent
Cluster 1
cloud
Host 1
Primary
Storage
Host 2

16. CloudStack Cloud Architecture
Cloud
Data Center 1 Data Center 2
Data Center 2
Data Center 3
Zone 2
Zone 2
Zone1 Zone 3
Zone 4 3
Zone
CloudStack Cloud can have
one or more Availability
Zones (AZ).
Data Center 2
Data Center 2
Data Center 2
Zone 2
Zone 2
ZoneZone 3
2
Zone 3
Zone 3
- Do Not Distribute

17. Management Server Managing
Multiple Zones
Cloud
Data Center 1 Data Center 2  Single Management Server can
Data Center 2
Managem Data Center 3 manage multiple zones
ent
Server Zone 2  Zones can be geographically
Zone 2 distributed but low latency links are
Zone 3 expected for better performance
Zone1
Zone 4 3
Zone
 Single MS node can manage up to
5K hosts.
 Multiple MS nodes can be deployed
Data Center 2 as cluster for scale or redundancy
Data Center 2
Data Center 2
Zone 2
Zone 2
Zone Zone 3
2
Zone 3
Zone 3
- Do Not Distribute

18. Management Server Deployment
Architecture
Single-node Multi-node
Deployment Deployment
Managem
ent
Server
User API User API
Managem Managem
ent MySQL Load
ent
Server DB Balancer
Server
Admin API Admin API
Managem MySQL
ent DB
Server
Back Up
Replication DB
 MS is stateless. MS can be deployed
as physical server or VM
 Single MS node can manage up to Infrastructure
Infrastructure
10K hosts. Multiple nodes can be
Resources Resources
deployed for scale or redundancy
 Commercial: RHEL 5.4+; FOSS:
Ubuntu 10.0.4, Fedora 16
- Do Not Distribute

19. CloudStack Storage
Primary Storage
• Configured at Cluster-level. Close to hosts for better
performance
L3 switch
• Stores all disk volumes for VMs in a cluster
• Cluster can have one or more primary storages
Pod 1 L2 switch
• Local disk, iSCSI, FC or NFS Secondary
Cluster 1 Storage
Host 1
Primary
Secondary Storage Storage
Host 2
• Configured at Zone-level
• Stores all Templates, ISOs and Snapshots
• Zone can have one or more secondary storages
• NFS, OpenStack Swift
- Do Not Distribute

20. Core CloudStack Components
VM
• Hosts
• Servers onto which services will be provisioned Host
VM
• Primary Storage Network
• VM storage Host
• Cluster
Primary
• A grouping of hosts and their associated storage Storage
• Pod
• Collection of clusters Cluster
• Network
Secondary
• Logical network associated with service offerings Storage Cluster
• Secondary Storage
• Template, snapshot and ISO storage CloudStack Pod
• Zone
• Collection of pods, network offerings and secondary
storage CloudStack Pod
• Management Server Farm
• Responsible for all management and provisioning tasks
Zone

21. Understanding the Role of Storage and Templates
• Primary Storage
• Cluster level storage for VMs
Host
• Connected directly to hosts
• NFS, iSCSI, FC and Local Host
• Secondary Storage Primary Storage
• Zone level storage for template, ISOs and Cluster
snapshots
• NFS or OpenStack Swift via CloudStack Pod
System VM
• Templates and ISOs
• Imported into CloudStack
• Can be private or public Secondary Storage
Zone
Template

22. Provisioning Process
1. User Requests Instance VM
2. Provision Optional Network Services Host
3. Copy instance template from Host
secondary storage to primary storage Primary Storage
on appropriate cluster
Cluster
4. Create any requested data volumes on
primary storage for the cluster Pod
5. Create instance Template
6. Start instance
Secondary Storage
Zone

23. Citrix XenServer
CloudStack
• Integrates directly with XenServer Manager
Pool Master
• Snapshots at host level XenServer Pool
Master Host
• System VM control channel at host
level XenServer Host
• Network management is host level XenServer Host
XenServer Host
XenServer Host
XenServer
Resource Pool

24. Oracle VM
CloudStack
• Integrates with ovs-agent Manager
• Snapshots at host level
OVS Agent
• System VM control channel at OVM Host
host level
OVS Agent
• Network management is host OVM Host
level
OVS Agent
• Does not use OVM Manager
OVM Host
• All templates must be from Oracle
OVS Agent
• CloudStack configures ocfs2 nodes OVM Host
• Requires “helper” cluster

25. RedHat Enterprise Linux (KVM)
• Integrates with libvirt using Cloud
CloudStack
Agent Manager
• Snapshots at host level
Cloud Agent
• System VM control channel at
host level Libvirt
KVM Host
• Network management is host
level
Cloud Agent
• Only RHEL 6, not RHEV
Libvirt
• Also supports Ubuntu 10.04
KVM Host

26. VMware vSphere
CloudStack
• Integration through vCenter Manager
vSphere Host
• System VM control channel via
vCenter
CloudStack private network vSphere Host
• Snapshot and volume vSphere Cluster
management via Secondary
Storage VM vSphere Host
• Networking via vSphere vSwitch vSphere Host
vSphere Host
vSphere Cluster
Data Center

27. Management Server Interaction with Hypervisors
Managem
ent Server
XAPI HTTP
vCenter Agent Agent
XenServer
KVM OVM
ESX
• XS 5.6, 5.6FP1, 5.6 SP2, 6.0 • ESX 4.1, 5.0 (coming) • RHEL 6.0, 6.1, 6.2 (coming) • OVM 2.2
• Incremental Snapshots • Full Snapshots • Full Snapshots (not live) • No Snapshots
• VHD • VMDK • QCOW2 • RAW
• NFS, iSCSI, FC & Local disk • NFS, iSCSI, FC & Local disk • NFS, iSCSI & FC • NFS & iSCSi
• Storage over-provisioning: • Storage over-provisioning: • Storage over-provisioning: • No storage over-
NFS NFS, iSCSI NFS provisioning

28. Multi-tenancy & Account Management
Cloud
Resources
Domain
VMs, IPs,
Snapshots…
• Domain is a unit of
Org A isolation that represents
Admin a customer org, business
unit or a reseller
Domain
Reseller A
• Domain can have
Admin Resources arbitrary levels of sub-
Sub-Domain VMs, IPs,
Org C
Snapshots… domains
Admin
• A Domain can have one
Account
or more accounts
Group A
• An Account represents
Account one or more users and is
Group B the basic unit of isolation
User 1 • Admin can limit
resources at the Account
User 2
or Domain levels

29. Physical Network
Operations
Users
Admin and
Cloud API
CloudStack
Mgmt Server
Cluster Router
MySQL
Load Balancer
Availability Zone
L3 Core Switch
Access
Layer
Switches
Secondary
Servers
… … … … … Storage
Pod 1 Pod 2 Pod 3 Pod N

30. Layer 3 cloud networking
Web DB Web
VM VM VM
Web DB
Security Security
Group Group
Web Web DB
VM VM VM
… … …
Web Web
VM VM

31. Guest Networks with L3 isolation
Public Public IP Guest Guest
Internet address 1 VM address
65.37.141.11 1 10.1.0.2
10.1.0.1 Guest
65.37.141.24 Pod 1 L2 Guest
65.37.141.36 Switch 2 VM address
65.37.141.80 1 10.1.0.3
Guest Guest
1 VM address
L3 Core
Pod 2 L2
Switch
10.1.8.1
… 2 10.1.0.4
Switch
Guest Guest
Load 10.1.16. 2 VM address
Pod 3 L2
Balancer 1 2 10.1.16.12
Switch
Guest
Guest
2 VM
address
3
10.1.16.21
… Guest
1 VM
Guest
address
3
10.1.16.47
Guest
Guest
1 VM
address
4
10.1.16.85

32. Virtual Networks (L2 isolation)
Core (L3) Network
Pod K Pod M Pod N
Access Switch(es) V
Hypervisor
V
V
Hypervisor
R
…
CLUSTER 1
Hypervisor 1
R
VM Traffic …
Hypervisor 8
Public Traffic
…
CLUSTER 4
V V
Hypervisor N
V Tenant VM
Hypervisor N+1
V
R Tenant Virtual Router

33. Guest virtual layer-2 network
Guest Virtual Network
10.1.1.0/24
Public Public IP Guest
Gateway Guest
Network address 1 VM
address address
65.37.141.11 1
10.1.1.1 10.1.1.2
65.37.141.36
Guest 1 Guest Guest
Public Virtual 1 VM address
Internet Router 2 10.1.1.3
NAT
Guest Guest
DHCP
1 VM address
Load
3 10.1.1.4
Balancing
VPN Guest Guest
1 VM address
4 10.1.1.5
Guest Virtual Network
Public IP 10.1.1.0/24
address Gateway Guest Guest
65.37.141.24 address 2 VM address
65.37.141.80 10.1.1.1 1 10.1.1.2
Guest 2 Guest Guest
Virtual 2 VM address
Router 2 10.1.1.3
NAT
Guest Guest
DHCP
2 VM address
Load
3 10.1.1.4
Balancing
VPN

34. Layer-2 Guest Virtual Network
CS Virtual Router provides Network Services External Devices provide Network Services
Guest Virtual Network 10.1.1.1/8 Guest Virtual Network 10.1.1.1/8
VLAN 100 VLAN 100
Public Public
Network/Intern Network/Intern
et Guest et Guest
Public IP Private IP 10.1.1.1
10.1.1.1 VM 1 10.1.1.111 VM 1
Gateway 65.37.141.11 Juniper
Public IP 1 SRX
address
65.37.141.11 CS Firewall
10.1.1.1 Guest Guest
Virtual
10.1.1.3 VM 2 10.1.1.3 VM 2
Router
Public IP Private IP
DHCP, DNS 65.37.141. NetScaler 10.1.1.112
NAT Guest 112 Load Guest
Load Balancing 10.1.1.4 VM 3 Blancer VM 3
10.1.1.4
VPN
Guest Guest
10.1.1.5 VM 4 10.1.1.5 VM 4
CS
DHCP, Virtual
Router
DNS

35. Layer-3 Guest Network
Network Services Managed Externally Network Services Managed by CS
Public Network
65.11.0.0/16
Security Group
Security Group
Public 1
1
Network/Intern 10.1.2.3
65.11.1.2 et Guest VM
Guest VM
1
1
65.11.1.2 NetScal L3
10.2.12.4
65.11.1.3 Guest VM
Guest VM er switc
65.11.1.3 2
2 Load h
65.11.1.4 Blancer
EIP,
ELB 10.5.2.99
65.11.1.4 Guest VM
Guest VM
3
3
10.1.2.18
65.11.1.5 Guest VM
Guest VM
4
4
CS
CS Virtual
Virtual DHCP, Security Group
DHCP, Security Group Router
Router DNS 2
DNS 2

36. Network Offerings
• Cloud provider defines the
feature set for guest networks
• Toggle features or service levels
– Security groups on/off
– Load balancer on/off
– Load balancer software/hardware
– VPN, firewall, port forwarding
• User chooses network offering
when creating network
• Enables upgrade between
network offerings
• Default offerings built-in
– For classic CloudStack networking

37. CloudStack System VMs
• System VMs optimize and scale the datapath on behalf of CloudStack
– Stateless, can be destroyed and recreated from database state
– Highly Available
– Communicates with Management Server over management network
– Usually have 3 interfaces: control, guest and public
• Console Proxy VM
– Provides AJAX-style HTTP-only console viewer
– Grabs VNC output from hypervisor
– Scales out (more spawned) as load increases
– Java-based server Communicates with MS over message bus
• Secondary Storage VM
– Provides image (template) management services
– Download from HTTP file share or Swift
– Copy between zones
– Scale out to handle multiple NFS mounts
– Java-based server communicates with MS over message bus

38. CloudStack System VMs
• Virtual Router VM
– Provides multiple network services
– IPAM (DHCP), DNS, NAT, Source NAT, Firewall, PF, VPN
– User-data, Meta-data, SSH keys and password change server
– Redundancy via VRRP
– MS configures VR over SSH
• Proxied via the hypervisor on XS and KVM

39. Virtual Router Information (applies to
all Sys. VMs)
• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security
APT repository. No extraneous accounts
• 32-bit for enhanced performance on Xen/VMWare
• Only essential software packages are installed. Services such as, printing, ftp, telnet, X, kudzu,
dns, sendmail are not installed.
• SSHd only listens on the private/link-local interface. SSH port has been changed to a non-
standard port (3922). SSH logins only using keys (keys are generated at install time and are
unique for every customer)
• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum
performance on all hypervisors. Xen tools inclusion allows performance monitoring
• Template is built from scratch and is not polluted with any old logs or history
• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved
security and speed
• Latest version of jre from Sun/Oracle ensures improved security and speed

40. System VM contd
• SSH keys and password are unique to cloud
installation
• Code can be patched by restarting system vm
– Mounts a special ISO file with latest code at boot
– If ISO contents differ, patch and reboot
• Same system vm works on XS, KVM, VMWare
– Bootstrap step for the cloud is to install the template
for this system vm
• Ready to be re-purposed for other specialized
tasks

41. Architecture Components
Service Management (Billing, Metering, Accounts, etc.)
User Interface Image Libraries
Maintenance and Provisioning
Administrato
Operation, Administration,
End User Console Application Catalog
r
Integration API
Custom Templates
Developer API
Operating System ISOs
Amazon OpenStack Custom
Availability and Security
Backup Load Balancing High Availability Monitoring
Resource Management Dynamic Workload Management
Virtualization Layer
Compute Network Storage

42. Interactions
OVM Cluster Primary
Storage
vcenter
Monitoring Primary
CS API vSphere Cluster
Storage
End
User UI
Primary
XS Cluster Storage
Admin
UI
Clustered
CloudStack XAPI
Domain CS Admin & CloudStack
CloudStack
Admin End-user API Primary
UI
Management JSON KVM Cluster Storage
Server
NetConf
Juniper SRX
Cloud user Nitro API
{API client (Fog/etc)} VNC
JSON
ec2 API JSON Netscaler
Cloud user Console
Console
{ec2 API client } Proxy VM
Proxy VM NFS
MySQL Server
{Proxied} SSH Sec. Storage NFS NFS
Sec. Storage
VM
Ajax HTTPS VM
Console
Router VM HTTP (Template Download)
Router VM HTTP (Template Copy)
Router VM
Cloud user HTTP (Swift)

43. Inside a Management Server
Plugins
cmd.execute() Plugins
CloudStack Commands
Async Plugins
API API Job
Ser Queu Serv
vlet e ices Kernel
Responses API
Mgr
Agent
API Messa Resources
(Cmds) ge Bus Local
Or
Remote
Agent
Manager
Hypervisor Network
Native Device
APIs API
MySQL