The document discusses high availability (HA) techniques in OpenStack. It covers HA concepts for both stateless and stateful services. For compute HA, it discusses server evacuation and instance migration without and with shared storage. It then covers different HA options for OpenStack controllers, including Pacemaker/Corosync/DRBD for active-passive HA and Galera for active-active MySQL HA. It also discusses using Keepalived, HAProxy and VRRP for load balancing and failover of API services. Finally, it presents a sample highly available OpenStack architecture and lists additional resources.
2. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Agenda
• What is HA
• Compute HA
• Controller HA
• Corosync, Pacemaker and DRBD
• Galera
• HAProxy, keepalived, VRRP
• Resources and Summary
2
4. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability
4
Minimize data loss
Minimize system downtime
5. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability Concepts
• Stateless services
– There is no dependency between requests
– For example: Nova API, Nova Scheduler, etc.
• Stateful services
– An action typically comprises multiple requests
– For example: MySQL, RabbitMQ, etc.
5
6. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability Concepts
• Active/Passive
– Redundant instances of stateless services are load balanced
– For Stateful services a replacement resource can be brought online.
• Active/Active
– Redundant instances of stateless services are load balanced
– Stateful services are managed in such a way that services are redundant, and that all
instances have an identical state.
– Updates to one instance of a database would also update all other instances.
6
9. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Server Evacuation
9
Evacuation
Evacuation
• Without Shared Storage
– The instance will be booted from a
new disk, but will preserve the
configuration, e.g. id, name, uid,
ip...etc.
• With Shared Storage
– The instance will be booted from
same disk and data will be preserved
11. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
11
Virtualization vs. Cloud
• Virtualization needs care and feeding
– Name the VM
– Tune and groom regularly
– Feed it with good food and supplements
– Take to the vet when sick
• Cloud servers are disposable
– VMs are not unique
– Tune and groom apps not the cows
– Keep the cow upright
– Shoot the cow when it is sick
12. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Scale Up vs. Scale Out
12
Traditional Cloud
15. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Pacemaker, Corosync and DRBD
15
• Pacemaker
– high availability and load balancing stack for
the Linux platform.
– Interacts with applications through Resource
Agents (RA)
• Corosync
– Totem single-ring ordering and membership
protocol
– UDP and InfiniBand based messaging,
quorum, and cluster membership to
Pacemaker.
• DRBD (Distributed Replication Block
Device)
– Synchronizes Data at the block device
– Uses a journaling system (such as ext3 or
ext4)
16. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Galera
• Synchronous multi-master cluster
technology for MySQL/InnoDB
– MySQL patched for wsrep (Write Set
REPlication)
– Active/active multi-master topology
– Read and write to any cluster node
– True parallel replication, in row level
– No slave lag or integrity issues
16
18. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Keepalived, HAProxy and VRRP
• HAProxy
– Load Balancing and Proxying for HTTP and TCP Applications
– Works over multiple connections
– Used to load balance API services
• VRRP (Virtual Router Redundancy Protocol)
– Eliminates SPOF in a static default routed environment
• Keepalived
– Based on Linux Virtual Server (IPVS) kernel module to provide layer 4 Load Balancing
– Implements a set of checkers to check service status and to maintain health
– Leverage the VRRP Protocol to remap VIPS in event of failure
18
19. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Sample OpenStack HA Architecture
19
.…
Availability
Zone 1
Dedicated Firewalls
BOND
0
BOND
1
BOND
0
BOND
1
Controller
API Services
API & Horizon
Cinder API
Nova Scheduler
Keystone
Glance
RabbitMQ
MYSQL
Chef
Server
Recipes
Load Balancers
Redundant Network Switches
Storage
EMC, NetApp, or
Solidfire
Vols
BOND
2
Redundant Network Switches
Inside LB VLAN
Storage Network (private)
Fixed Network (private)
Compute 1
KVM
G2
G1
G4
G3
Compute N
KVM
G6
G5
G7
BOND
0
BOND
1
BOND
2
.…
Availability
Zone 2
BOND
0
BOND
1
BOND
2
Compute 1
KVM
Compute N
KVM
G1
6
G15
G17
BOND
0
BOND
1
BOND
2
BOND
0
BOND
1
BOND
2
G12
G11
G14
G13
BOND
2
Controller
API Services
API & Horizon
Cinder API
Nova Scheduler
Keystone
Glance
RabbitMQ
MYSQL
Chef
Server
Recipes
22. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
22
Comparison
Database Replication
method
Strengths Weakness/Limita
tions
Keepalived/HAPro
xy/VRRP
Works on MySQL
master-master
replication
Simple to
implement and
understand.
Works for any
storage system.
Master-master
replication does
not work beyond 2
nodes.
Pacemaker/Coros
ync/DRBD
Mirroring on Block
Devices
Well tested More complex to
setup. Split Brain
possibility
Galera Based on write-
set Replication
(wsrep)
No Slave lag Needs at least 3
nodes. Relatively
new.
Others MySQL Cluster,
RHCS with
DAS/SAN storage
Well tested More complex
setup.
24. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Resources
• OpenStack
– openstack.org
– launchpad.net/openstack
– #openstack
– #openstack on webchat.freenode.net
• OpenStack HA
– http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf
– https://github.com/rcbops-cookbooks/
• MySQL HA
– http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-
deployment-guide/
– http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html
– https://www.hastexo.com/
– http://www.drbd.org/
25. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
2
5
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
For More Information
You can reach me at:
Kenneth Hui
Open Cloud Architect
Rackspace
E-mail: ken.hui@rackspace.com
Twitter: @hui_kenneth
Blog: http://cloudarchitectmusings.com
I work as an Open Cloud Architect with Rackspace. Rackspace has the distinction of being one of the founders of OpenStack and the proprietor of the largest OpenStack-based Public Clouds in the world. As such, we have a vested desire and interest in driving the adoption of OpenStack and the growth of it’s community.However, Rackspace does not own OpenStack and has in fact given it to the community via the Linux Foundation and the the Apache 2 license. In that same spirit, I am happy to help anyone who wishes to learn more about OpenStack, regardless of rather you are now or ever will be a Rackspace customer. So please feel free to contact me and to ask questions or to have me talk about OpenStack with your company or user group. I tweet frequently about OpenStack and maintain a blog that often has OpenStack and/or Cloud Computing related posts.