Masakari and recent activity in openstack HA team. This slide is presented at Japan OpenStack User Group on 24th Mar, 2016. 日本OpenStackユーザ会 第25回 の資料。OpenStack インスタンス HA 機能の Masakari と HA team の活動紹介

1. © 2016 NTT Software Innovation Center
Masakari と HA コミュニティの状況
2016/03/24
室井 雅仁

2. 2Copyright©2016 NTT corp. All Rights Reserved.
自己紹介
• 室井 雅仁 (むろい まさひと)
• 所属: NTT
• OpenStackを利用したOSSクラウドのアーキテクトを担当
• 社内向け OpenStack 環境の運用、コミュニティへのアップスト
リーム
• OpenStack Congress Core Reviewer
• https://wiki.openstack.org/wiki/Congress
• https://thinkit.co.jp/article/8381
• IRC: masahito
• エディタ: Emacs 派

3. 3Copyright©2016 NTT corp. All Rights Reserved.
http://www.slideshare.net/masahito12/openstack-
masakari

4. 4Copyright©2016 NTT corp. All Rights Reserved.
本日のお品書き
• Masakari
• OpenStack HA コミュニティの現状
• Masakari デモ (時間があれば)

5. 5Copyright©2016 NTT corp. All Rights Reserved.
Masakari

6. 6Copyright©2016 NTT corp. All Rights Reserved.
What’s Masakari
Copyright © いらすとや. All Rights Reserved.
まさかり

7. 7Copyright©2016 NTT corp. All Rights Reserved.
What’s Masakari in OpenStack
• Virtual Machine High Availability (VM-HA) service
• Rescue Virtual Machine when any errors occur
• Published as OSS at github https://github.com/ntt-sic/masakari

8. 8Copyright©2016 NTT corp. All Rights Reserved.
Motivations
• Pets vs Cattle
• Unable to change all Apps to Cloud Native at once
• Open Source

9. 9Copyright©2016 NTT corp. All Rights Reserved.
Requirements for Pets Model
• Detect 3 types of VM down
• VM instance down
• VM manager down
• Host down
• Recover VM within 5 mins
• Work automatically

10. 10Copyright©2016 NTT corp. All Rights Reserved.
Architecture Overview
ComputeNodesControllerNodes
&BackendNodes

11. 11Copyright©2016 NTT corp. All Rights Reserved.
How to detect the 3 down
• VM down
• Monitoring libvert’s events
• Manager Process down
• Monitoring manager process
• Host down
• Monitoring Pacemaker status

12. 12Copyright©2016 NTT corp. All Rights Reserved.
Detect VM Down
Libvirt
Masakari
1. Notify down VM’s Info
(VM-ID, Host Name, etc.)
Libvirt Monitor
Detect VM down
VM1 VM2 VM3
Libvirt
Libvirt Monitor
VM5 VM6
HostHost
Nova
2. Call Rebuild API for the down VM
3. Rebuild the VM
Down
0. Set callbacks for
libvert’s events hook

13. 13Copyright©2016 NTT corp. All Rights Reserved.
Manager Process Down
1. Restart manager
process when it’s down
Process Monitor
Masakari
2. Notify manager process down
if fail to restart few times
Libvirt Nova-compute
Host A
Libvirt Nova-compute
Host B
Nova
3. Notify Nova to disable schedule
for Host A
Process Monitor
Down

14. 14Copyright©2016 NTT corp. All Rights Reserved.
Host Down
RA
CIB
RA
RA
Node’s
Status
pacemaker
Heartbeat communications or
pacemaker-remote
Masakari
Check its Host’s status
1. Notify another host down
Start
Stop
Monitor
WatchDog&
Shutdowner
Host Fail Monitor
Polling
RA
CIB
RA
RA
Node’s
Status
pacemaker
Start
Stop
Monitor
WatchDog&
Shutdowner
Host Fail Monitor
Polling
Down
Host A Host B
Nova
2. Call Evacuate API for all VM on Host B

15. 15Copyright©2016 NTT corp. All Rights Reserved.
How to use Masakari
1. Prerequisites
• Set up Nova and Compute Nodes with KVM
• Set up a shared storage per cluster for ephemeral disks (e.g. NFS)
2. Install and Configure Masakari
• Download source from github
• https://github.com/ntt-sic/masakari
• Install Masakari’s package
• Initialize Masakari’s DB
• Configure 4 Masakari’s config files
3. Start Masakari
• Start all process
• Add a reserved host prepared for host down
4. Wait any error
• Masakari only works when any error occurs

16. 16Copyright©2016 NTT corp. All Rights Reserved.
Run Masakari in local
• Run masakari-deploy!!
http://github.com/ntt-sic/masakari-deploy
• For quick QA or demo

17. 17Copyright©2016 NTT corp. All Rights Reserved.
Challenges
• No branch from OpenStack master

18. 18Copyright©2016 NTT corp. All Rights Reserved.
Masakari Next Release
• 1.1.0 is supposed be released until Austin
Summit
• Supports Pacemaker-Remote for hostmonitor
• Supports PostgreSQL and other DB
• By using SQLAlchemy
• Supports CentOS
• With python installer and rpm packaging
• Uses python-client to call OpenStack API

19. 19Copyright©2016 NTT corp. All Rights Reserved.
OpenStack HA コミュニティの現状

20. 20Copyright©2016 NTT corp. All Rights Reserved.
OpenStack HA チーム
• openstack-ha チーム
• VM HA のユースケース議論や今後の実装方針考えるチーム
• IRC チャンネル: #openstack-ha
• IRC ミーティング: 月曜 18:00 #openstack-meeting
• Etherpad:
https://etherpad.openstack.org/p/automatic-evacuation
• 参加メンバ(企業)
• NTT, RedHat, SUSE, Intel, ...
ググっても多分みつかりません。
チーム名が一般的過ぎて別ページがヒットします。

21. 21Copyright©2016 NTT corp. All Rights Reserved.
チームで議論中の VM HA 方式
1. Masakari
• https://github.com/ntt-sic/masakari
2. Resource-agent
• https://github.com/openstack/openstack-
resource-agents
3. Mistral workflow
• https://github.com/gryf/mistral-evacuate
• Mistral プロジェクトを利用した VM-HA フロー
• Mistral: ワークフローエンジン as a Service
• https://wiki.openstack.org/wiki/Mistral

22. 22Copyright©2016 NTT corp. All Rights Reserved.
コミュニティ内関係図
VM HA 機能
Masakari
Resource-
agent
Mistral
workflow
HA VM
user story
大
Product WG
執筆
……
VM HA 機能の既存実装
大
HA チーム
HA チームと
Product WG
で連携しよう
今後の方向性を議論中
あるべき姿の議論中

23. 23Copyright©2016 NTT corp. All Rights Reserved.
コミュニティの今後の展望
• コミュニティとしての VM HA の仕様・実装
• Product WG と協力
• 各種 VM HA の実装の方向性議論
• Masakari と Mistral のインテグレーションなど
• Austin Summit でチームのサマリーに関するセッションも有
ります
• High Availability for pets and hypervisors - State of the Nation
(https://www.openstack.org/summit/austin-2016/summit-
schedule/events/7327)
• コミュニティでの議論状況の共有が主な話題
• OpenStack HA チームの次回作に乞うご期待!!

24. 24Copyright©2016 NTT corp. All Rights Reserved.
Masakari デモ

25. 25Copyright©2016 NTT corp. All Rights Reserved.
デモシナリオ
1. VM Instance down
Host1 Host2
VM
down
1.vm down
2.restart
Controller
Nova Masakari
Host1 Host2
VM
down
1.host down
2.evacuate
Controller
Nova Masakari
2. Host down

26. 26Copyright©2016 NTT corp. All Rights Reserved.
Github: https://github.com/ntt-sic/masakari
Mail: muroi.masahito@lab.ntt.co.jp
Pull Request 気軽にどうぞ
日本語でも OK です。