1. Copyright©2015 NTT corp. All Rights Reserved.
Matcha: fully HA configured
OpenStack
Masahito Muroi, NTT
IRC: masahito
muroi.mashaito@lab.ntt.co.jp
2. 2Copyright©2015 NTT corp. All Rights Reserved.
• Bitter Japanese Green Tea
• Used for Japanese Tea ceremony
• Many kind of snacks have Matcha flavor
• Try it in Tokyo!
What’s Matcha
Photo by (c)Tomo.Yun, http://www.yunphoto.net
3. 3Copyright©2015 NTT corp. All Rights Reserved.
•Fully HA configured OpenStack
•Having VM-HA feature
•Also having full scalability
What’s Matcha
Controller Node(2)
pacemaker(1Act-1Sby)
•VIP(neutron-sv, haproxy)
•neutron-server
•nova-consoleauth
•Masakari
keystone-all
nova-api
nova-conductor
nova-novncproxy
nova-scheduler
cinder-api
cinder-scheduler
Apache(keystone)
Haproxy
Network Node(4)
Compute Node(4)
nova-compute
neutron-ovs-agent
libvirt_monitor
rc_host_monitor
rc_process_monitor
Backend Node(3)
mysql-pxc(3Act)
RabbitMQ(2Act)
pacemaker(nAct-1Sby)
• neutron-linuxbridge-agent
• neutron-dhcp-agent
• neutron-l3-agent
pacemaker(nAct)
Storage Node(2)
glance-api
glance-registry
Apache(API Filter)
pacemaker(nAct-1Sby)
•cinder-volume(NFS)
pacemaker(3Act)
•VIP(MQ, PXC)
4. 4Copyright©2015 NTT corp. All Rights Reserved.
•Multi Active
•Access through VIP controlled by
Pacemaker
Controller Node HA
pacemaker(1Act-1Sby)
•VIP(neutron-sv, haproxy)
•neutron-server
•nova-consoleauth
•Masakari
keystone-all
nova-api
nova-conductor
nova-novncproxy
nova-scheduler
cinder-api
cinder-scheduler
Apache(keystone)
Haproxy
pacemaker(1Act-1Sby)
•VIP(neutron-sv, haproxy)
•neutron-server
•nova-consoleauth
•Masakari
keystone-all
nova-api
nova-conductor
nova-novncproxy
nova-scheduler
cinder-api
cinder-scheduler
Apache(keystone)
Haproxy
5. 5Copyright©2015 NTT corp. All Rights Reserved.
•Multi Active
•RabbitMQ using mirrored queue
•MySQL with Percona-cluster backend
•Access through VIP controlled by
Pacemaker
Backend Node HA
mysql-pxc(3Act)
RabbitMQ(2Act)
pacemaker(3Act)
•VIP(MQ, PXC)
mysql-pxc(3Act)
RabbitMQ(2Act)
pacemaker(3Act)
•VIP(MQ, PXC)
mysql-pxc(3Act)
RabbitMQ(2Act)
pacemaker(3Act)
•VIP(MQ, PXC)
6. 6Copyright©2015 NTT corp. All Rights Reserved.
•Active-Stanby
•Controlled by Pacemaker
Storage and Network Node HA
Storage Node(2)
glance-api
glance-registry
pacemaker(nAct-1Sby)
•cinder-volume(NFS)
Network Node(4)
pacemaker(nAct-1Sby)
•neutron-linuxbridge-agent
•neutron-dhcp-agent
•neutron-l3-agent
glance-api
glance-registry
pacemaker(nAct-1Sby)
•cinder-volume(NFS)
pacemaker(nAct-1Sby)
•neutron-linuxbridge-agent
•neutron-dhcp-agent
•neutron-l3-agent
7. 7Copyright©2015 NTT corp. All Rights Reserved.
•Multi Active
•Masakari [1] for VM-HA
•can rescue VM down
•can rescue nova-compute down
•can rescue Hypervisor down
Compute Node HA
nova-compute
neutron-ovs-agent
libvirt_monitor
rc_host_monitor
rc_process_monitor
pacemaker(nAct)
[1]: https://github.com/ntt-sic/masakari
nova-compute
neutron-ovs-agent
libvirt_monitor
rc_host_monitor
rc_process_monitor
pacemaker(nAct)
nova-compute
neutron-ovs-agent
libvirt_monitor
rc_host_monitor
rc_process_monitor
pacemaker(nAct)
8. 8Copyright©2015 NTT corp. All Rights Reserved.
• Service Model: Pets vs Cattle
• Cloud-native application handles High Availability
at its own layer. (Cattle service model)
• Sometimes, customers and/or applications still
prefer “Pets” service model.
• Masakari
• Provides VMHA for “Pets” service model
• Automatic VM instance recovery in case of VM or
hypervisor failure to minimize the downtime
• Requires Nova and Shared file system such as NFS
• Needs No modification of OpenStack components
• Uses Pacemaker to detect a host failure
• Is now at https://github.com/ntt-sic/masakari
• Author: Kentaro TANAKA
<tanaka.kentaro@lab.ntt.co.jp>
• Is mostly written in Python
• Is open source under Apache license
Masakari: VMHA for OpenStack Compute
9. 9Copyright©2015 NTT corp. All Rights Reserved.
Masakari: Architecture and Data Flow
OpenStack API
ComputeNodesControllerNodes
&BackendNodes
