A healthy diet for your Java application Devoxx France.pdf
Neutron high availability open stack architecture openstack israel event 2015
1. @Livnat_Peer
Sr. Engineering Manager, Red Hat
@ArthurBerezin
Sr. Technical Product Manager, Red Hat
Neutron High Availability
OpenStack Israel
Tel-Aviv June 2015
2. Agenda
HA Enabling Technologies
Pacemaker and HAProxy
Neutron Built-in Mechanisms
DHCP Agent HA
L3 Agent with
Virtual Router Redundancy Protocol(VRRP)
Distributed Virtual Routing(DVR)
5. High Availability
● Minimize Downtime By Avoiding SPOF
● Service redundancy
○ Active-Active When possible
■ Stateless services
■ Built-in HA mechanisms
○ Active-Passive for others
● Scale out Architecture
Add nodes as you go
8. ● Cluster Resource Manager
● Uses Corosync for cluster communication
● Monitor and Control Resources:
○ Floating Virtual IP Address (VIP)
○ SystemD/LSB/OCF Services
○ Cloned Services(Active/Active)
● STONITH - Fencing with Power Management
○ Important for ensuring data consistency
Pacemaker
9. ● Virtual IP(VIP)
● SystemD Cloned Resource
● STONITH Fencing
Pacemaker OpenStack Service
Node 2 - 192.168.1.2Node 1 - 192.168.1.1
pcsd pcsd
Cloned
STONITH STONITH
Service Service
Service
Virtual IP
10.0.0.1
10. HAProxy Load Balancer
Load Balancing and Proxy for HTTP/TCP
● Mature and popular with web applications
● Health Checking
● Load Distribution
11. ● Load Distribution
○ Round Robin,
○ Stick-Table
● API Isolation
● Failure Detection
Node 1
Node 2 Node 3
HAProxy Load Balancer
Service Service
HAProxy
21. ● External mechanisms
● Neutron built-in mechanisms
● Reference implementation vs. vendors code
My HA Solution
22. Architecture - Assuming Centralized Network Node
Compute Node
Controller Node
Network Node
Neutron server
MySQL server
Neutron server
Neutron serverRabbitmq server
Neutron server
OVS agent
OVS
OVS Agent
keepalived
Neutron serverOVS
DHCP agentDHCP Agent
Neutron serverMetadata Agent
Metadata Proxy
dnsmasq
Internet
External
Network
API
Network
Management Network
Data Network
L3 Agent
23. DHCP Agent
● IP address allocation is done by the Neutron server
● dnsmasq is used as a distribution mechanism of predefined allocations
● The DHCP protocol allows multiple DHCP servers to co-exist while serving the same pool
● Configuration in Neutron
neutron.conf :
dhcp_agents_per_network = X OVS Agent
Neutron serverOVS
DHCP agent
Neutron serverMetadata Agent
Metadata Proxy
dnsmasq
L3 Agent
keepalived
DHCP Agent
28. VRRP (Virtual Router Redundancy Protocol)
● Providing HA of the network’s default gateway
● Configuring default gateway as VIP + Virtual MAC
● Gratuitous ARP after failover
Sync Net
29. L3 HA Implementing VRRP
● Using keepalived which internally implements VRRP
● Creating a per tenant HA network, used for VRRP sync messages
● When HA router is created it is scheduled on multiple network nodes (Configurable)
● New in Kilo
– Report which network node is hosting the master instance
● On the work
– L3 HA + l2pop
– External interface tracking
– L3 HA+DVR
30. Traffic Flow 3-tier Application
Host 1
WWW
VM
Host 2
App
VM
Host 3
DB
VM
Network
Node
Virtual
Router
31. DVR – Distributed Virtual Router
● DVR is moving most of the routing to the compute node
– Isolating the failure domain of the network node
– Optimizing the network flow
● Traffic types
– East – West (Within the tenant, different networks)
– North – South with floating IP (VM to/from external network)
– North – South without floating IP (Based on SNAT)
Direct between
compute nodes
Through
network node
32. Architecture - Assuming DVR
Compute Node
Controller Node
Network Node
Neutron server
MySQL server
Neutron server
Neutron serverRabbitmq server
Internet
External
Network
API
Network
Management Network
Data Network
Network Node
OVS Agent
keepalived
Neutron serverOVS
DHCP agentDHCP Agent
Neutron serverMetadata Agent
Metadata Proxy
dnsmasq
L3 Agent
Neutron server
OVS agent
OVS
33. Architecture - Assuming DVR
Compute Node
Controller Node
Network Node
Neutron server
MySQL server
Neutron server
Neutron serverRabbitmq server
Internet
External
Network
API
Network
Management Network
Data Network
Network Node
OVS Agent
keepalived
Neutron serverOVS
DHCP agentDHCP Agent
Neutron serverMetadata Agent
Metadata Proxy
dnsmasq
L3 Agent
Neutron server
OVS agent
OVS
34. Architecture - Assuming DVR
Compute Node
Controller Node
Network Node
Neutron server
MySQL server
Neutron server
Neutron serverRabbitmq server
Neutron server
OVS agent
OVS
Internet
External
Network
API
Network
Management Network
Data Network
Compute Node
Neutron server
OVS agent
Neutron serverOVS
L3 agent
Neutron serverMetadata agent
Metadata Proxy
Network Node
OVS Agent
keepalived
Neutron serverOVS
DHCP agentDHCP Agent
Neutron serverMetadata Agent
Metadata Proxy
dnsmasq
L3 Agent
35. Summary
● No one stop shop
● Maximize the use of built-in solutions
– They are vendor neutral
– Highly maintained
– Widely documented
● Understand what you need, use the appropriate tools
– DVR vs VRRP
– What size is your deployment, maybe A/P is good enough...
● The more complicated the solution is the more likely it is to have bugs