Virtual IPs or floating IPs have long been the workhorse mechanism for providing high-availability for database systems, however floating IP addresses have several limitations that make it problematic in modern data centers and cloud environments, notably that it requires all members be in the same Layer-2 domain. consul is a strongly consistent way of providing high-availability services in Layer-3 environments and provides fail-over across different geographic regions. In this talk we will discuss the benefits, setup, and use of consul for fail-over of PostgreSQL, both in a local data center scenario and a geographic redundancy scenario where databases are split across multiple data centers.

1. PostgreSQL High-Availability and
Geographic Locality using consul

2. Sean Chittenden
Engineering, HashiCorp
@SeanChittenden
sean@hashicorp.com
https://keybase.io/seanc

3. Quick Demo

4. Consul Consul
dc2dc1
PostgreSQL
Follower
PostgreSQL
Leader
PostgreSQL
Follower

5. CONSUL
HASHICORP

6. HASHICORP
Key Value Store
HTTP API
Host & Service
Level Health
Checks
Datacenter Aware
Consul solves four central challenges with SOA
Service
Discovery
HTTP + DNS

7. HASHICORP
Consul Installation

8. HASHICORP
Overview
1. Introduction to Consul
2. Review of Consul
a. Architecture
b. Agent Functionality
c. Agent Configuration
d. Features
3. Further Reading

9. HASHICORP
Introduction

10. HASHICORP
Consul powers runtime orchestration

11. CONSUL
HASHICORP
1. Service discovery
2. Service registry
3. Key/value store
4. Health checks

12. HASHICORP
Glossary
Agent - Long-running daemon on every member of the Consul
cluster. The agent is able to run in either client or server mode.
Client - Agent that forwards all RPCs to a server and
participates in the LAN gossip pool.
Server - Agent that maintains cluster state, responds to RPC
queries, exchanges WAN gossip with other datacenters, and
forwards queries to leaders of remote datacenters.
Consensus - Agreement upon the elected leader

13. HASHICORP
Glossary
Gossip - Random node-to-node communication primarily over
UDP that provides membership, failure detection, and event
broadcast information to the cluster. Built on Serf. Consul has
both LAN and WAN Gossip.
Datacenter - Networking environment that is private, low latency,
and high bandwidth. A Consul cluster is run per datacenter, so its
important to have low latency for the gossip protocol.

14. HASHICORP
Consul vs. Other Software
- Opinionated framework for service discovery using DNS
or HTTP
- Scalable gossip system that links server nodes and clients
- Distributed health checking with edge triggered updates
- Globally aware with multi-datacenter support
- Operationally simple
- Incorporation into the HashiCorp ecosystem

15. HASHICORP
Architecture

16. HASHICORP
Single Datacenter
CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT
SERVER SERVER SERVER
REPLICATION REPLICATION
RPC
RPC
LAN GOSSIP

17. HASHICORP
Multi-Datacenter
CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT
SERVER SERVER SERVER
REPLICATION REPLICATION
RPC
RPC
LAN GOSSIP
SERVERSERVER SERVER
REPLICATION REPLICATION
WAN
GOSSIP

18. HASHICORP
Raft Introduction
~/src/raft/thesecretlivesofdata/raft
open index.html
~/src/raft/raftscope
open index.html

19. HASHICORP
TCP and UDP Ports
Client HTTP RPC
TCP/8500
DNS
TCP/8600
UDP/8600 LAN Gossip
TCP/8301
UDP/8301
LAN Gossip
TCP/8301
UDP/8301
RPC
TCP/8400
RPC
TCP/8400
WAN Gossip
TCP/8301
UDP/8301
Clients consul1.dc1
Client RPC
(HTTP)
DNS
TCP/8600
UDP/8600
Server RPC
TCP/8300
consulN.dc2 consul2.dc1

20. HASHICORP
Agent functionality (client or server)
- RPC, HTTP, DNS APIs
- Health Checks
- Event Execution
- Gossip Participation
- Membership
- Failure detection

21. HASHICORP
Agent functionality (server)
- State replication
- Query Handling
- Leader election
- WAN Gossip

22. HASHICORP
Failover via DNS

23. HASHICORP
DNS Failover
• Works across L3 boundaries
in LAN environments
• Works across L3 boundaries
in WAN environments
• Small TTLs
• Workload Distribution
• Clients cache DNS data
• Not subject to spanning-tree
• Requires TCP connections
be reset on failover
• Clients can cache stale DNS
data
Pro Con

24. HASHICORP
Consul Installation

25. HASHICORP
consul Server 1/3
% cat config.json
{
"acl_datacenter": "lab1",
"acl_default_policy": "deny",
"acl_master_token": "rootToken",
"addresses": {
"dns": "0.0.0.0",
"http": "unix:///tmp/.consul.http.sock",
"https": "0.0.0.0",
"rpc": "unix:///tmp/.consul.rpc.sock"
},
"bootstrap_expect": 3,
"datacenter": "lab1",
"data_dir": "./svc/data",
"disable_remote_exec": true,

26. HASHICORP
Consul Server 2/3
"dns_config": {
"allow_stale": true,
"max_stale": "10080m",
"node_ttl": "60s",
"service_ttl": {
"*": "5s",
"stable-service": "86400s"
}
},
"encrypt": "[ random mime encoded data ]",
"log_level": "debug",
"ports": {
"https": -1
},
"server": true,
"unix_sockets": {
"mode": "0700"
}
}

27. HASHICORP
Consul Server 3/3
% cat svc/run
#!/bin/sh --
set -e
exec 2>&1
exec
/usr/bin/env -i
./bin/consul agent
-config-file=./config.json
-config-dir=./conf.d/
% cat svc/log/run
#!/bin/sh —
set -e
set 2>&1
exec chpst -u _log:_log svlogd ./main

28. HASHICORP
Consul Cluster
% consul members
Node Address Status Type Build Protocol DC
vm1 172.16.139.140:8301 alive server 0.7.0dev 2 lab1
% consul join 172.16.139.139 172.16.139.138
Successfully joined cluster by contacting 2 nodes.
% consul members
Node Address Status Type Build Protocol DC
vm1 172.16.139.140:8301 alive server 0.7.0dev 2 lab1
vm2 172.16.139.138:8301 alive server 0.7.0dev 2 lab1
vm3 172.16.139.139:8301 alive server 0.7.0dev 2 lab1

29. HASHICORP
Consul Cluster
% consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease = dev
revision = 'fa26d5f
version = 0.7.0
consul:
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = 172.16.139.139:8300
server = true
[snip]

30. HASHICORP
Consul Cluster
% consul info
[snip]
raft:
applied_index = 103339
commit_index = 103339
fsm_pending = 0
last_contact = 82.95803ms
last_log_index = 103339
last_log_term = 50663
last_snapshot_index = 98437
last_snapshot_term = 2228
num_peers = 2
raft_peers =
172.16.139.139:8300,172.16.139.138:8300,172.16.139.140:8300
state = Follower
term = 50663
[snip]

31. HASHICORP
dnsmasq Config
% cat /usr/local/etc/dnsmasq.conf
local-service
port=53
server=/consul/127.0.0.1#8600
rev-server=172.16.0.0/12,127.0.0.1#8600
server=208.67.222.222
server=208.67.220.220
cache-size=65536
% cat /etc/resov.conf
search localdomain
nameserver 127.0.0.1

32. HASHICORP
Service Discovery
HTTP + DNS

33. HASHICORP
- Nodes, Services, Checks
- Simple registration (JSON)
- DNS Interface
- HTTP API
Service Discovery

34. HASHICORP
PostgreSQL Service
% hostname
pg002
% cat config.d/pg-db.json
{
"service": {
"name": "pg-db",
"tags": ["follower"],
"port": 5432,
"checks":
[{
"id": "pg-alive",
"notes": "Make sure connect and queries work",
"script": "/usr/local/bin/check_postgresql",
"interval": "10s"
}]
}
}

35. Terminal
HASHICORP
$ dig follower.pg-db.service.consul

36. Terminal
HASHICORP
$ dig follower.pg-db.service.consul
; <<>> DiG 9.8.3-P1 <<>> follower.pg-db.service.consul
; (3 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 946
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0,
ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;follower.pg-db.service.consul. IN A
;; ANSWER SECTION:
follower.pg-db.service.consul. 0 IN A 172.16.139.141

37. Terminal
HASHICORP
$ dig follower.pg-db.service.consul SRV
; <<>> DiG 9.8.3-P1 <<>> follower.pg-db.service.consul
SRV
; (3 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 480
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0,
ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;follower.pg-db.service.consul. IN SRV
;; ANSWER SECTION:
follower.pg-db.service.consul. 0 IN SRV 1 1 5432

38. HASHICORP
DNS Interface
- Zero Touch
- Randomized Round-Robin DNS
- Filters on Health Checks

39. HASHICORP
HTTP API
- HTTP API
- Custom Integrations

40. HASHICORP
Host & Service Level
Health Checks

41. HASHICORP
What is a health check?
0PASSING
1WARNING
_
_FAILING
Any command that returns an exit code

42. HASHICORP
Health Checks & Monitoring
- Nagios-compatible
- Scalable
- Actionable
- Edge Triggered

43. Text Editor
HASHICORP
% cat conf.d/mem-check.json
{
"check": {
"id": "mem-util",
"name": "Memory utilization",
"script": "/usr/local/bin/mem_check.sh",
"interval": "10s"
}
}
Creating a check
Use a custom script

44. Text Editor
HASHICORP
% cat conf.d/http-check.json
{
"check": {
"id": "api",
"name": "HTTP API on port 4455",
"http": "http://localhost:4455/_health",
"interval": "10s",
"timeout": "1s"
}
}
Creating a check
Use a built-in check type

45. HASHICORP
Traditional Health Checking (pull)
HEALTH
CHECKING
SERVICE
DB 1
DB 2
DB N
"Are you healthy?"

46. HASHICORP
Traditional Health Checking (pull)
HEALTH
CHECKING
SERVICE
DB 1
DB 2
DB N
"Are you healthy?"
"Yessir!"

47. HASHICORP
Traditional Health Checking (pull)
HEALTH
CHECKING
SERVICE
DB 1
DB 2
DB N
"Are you healthy?"
"What about you?"
"Yessir!"

48. HASHICORP
Traditional Health Checking (pull)
HEALTH
CHECKING
SERVICE
DB 1
DB 2
DB N
"Are you healthy?"
"What about you?"
"Yessir!"
"Nah"

49. HASHICORP
Traditional Health Checking (pull)
DB 1
DB 2
DB N
HEALTH
CHECKING
SERVICE
"Are you healthy?"
"What about you?"
"Yessir!"
"Nah"

50. HASHICORP
Traditional Health Checking (pull)
DB 1
DB 2
DB N
HEALTH
CHECKING
SERVICE
1,000'S OF
REQUESTS

51. HASHICORP
Consul Health Checking (push)
CONSUL
DB 1
DB 2
DB N
My status has changed

52. HASHICORP
Consul Health Checking (push)
CONSUL
DB 1
DB 2
DB N
10'S OF
REQUESTS

53. HASHICORP
Liveness
- No Heartbeats
- Gossip-based Failure Detector built
on Serf
- Constant Load

54. HASHICORP
HTTP UI
http://172.16.139.138:8500/ui/#/lab1/services

55. HASHICORP
Key Value Store
HTTP API

56. Terminal
HASHICORP
$ curl -X PUT -d 'bar' http://localhost:8500/v1/kv/foo
true

57. Terminal
HASHICORP
$ curl -X PUT -d 'bar' http://localhost:8500/v1/kv/foo
true
$ curl http://localhost:8500/v1/kv/foo
[
{
"CreateIndex": 100,
"ModifyIndex": 200,
"Key": "foo",
"Flags": 0,
"Value": "YmFy"
}
]
% echo -n 'bar' | base64
YmFy
% echo -n 'YmFy' | base64 -d ; echo
bar

58. Terminal
HASHICORP
% cat <<EOF > acl.anonymous.json
{
"ID": "anonymous",
"Name": "Anonymous Token",
"Type": "client",
"Rules": "# Default all keys to read-only
key "" {
policy = "read"
}
# Default all services to read-only
service "" {
policy = "read"
}
# Allow hearing any user event by default.
event "" {
policy = "read"
}

59. Terminal
HASHICORP
# Default prepared queries to read-only.
query "" {
policy = "read"
}
# Read-only mode for the encryption keyring by default
(list only)
keyring = "read""
}
EOF
% curl -v -X PUT -d @acl.anonymous.json --unix-socket /
tmp/.consul.http.sock 'http://consul/v1/acl/update?
token=rootToken'

60. Prepared Queries

61. Use Case
• Multiple instances of a given service exist in
multiple datacenters
• Clients can talk to any of them, and always prefer
the instances with lowest latency
• Policies can change, desire to not have the clients
know the details of how to locate a healthy service

62. Prepared Queries
• New query namespace, similar to services
• Register queries to answer for parts of this
namespace
• Clients use APIs, or “.query.consul” DNS lookups
to run queries
• Magic happens :-)

63. pg-db with Failover
$ curl -X POST -d
'{
"Name": "geo-pg-db—follower",
"Service": {
"Service": "pg-db",
"Failover": {
"NearestN": 3
},
"Tags": ["follower"]
}
}’ localhost:8500/v1/query
geo-pgdb—follower.query.consul

64. PostgreSQL Template
$ curl -X POST -d
'{
"Name": "geo-db",
"Template": {
"Type": "name_prefix_match",
"Regexp": "^geo-db-(.*?)-([^-]+?)$"
},
"Service": {
"Service": "pg—${match(1)}",
"Failover": {
"NearestN": 3,
"Datacenters": ["dc1", "dc2"]
},
"OnlyPassing": true,
"Tags": ["${match(2)}"]
}
}' localhost:8500/v1/query
geo-db-customer-leader.query.consul
geo-db-customer-follower.query.consul
geo-db-billing-follower.query.consul
leader.pg-customer.service.consul
follower.pg-customer.service.consul
follower.pg-billing.service.consul

65. Catch All Template
$ curl -X POST -d
'{
"Name": "",
"Template": {
"Type": "name_prefix_match"
},
"Service": {
"Service": "${name.full}",
"Failover": {
"NearestN": 3
}
}
}' localhost:8500/v1/query
*.query.consul
With a single query template, all
services can fail over to the nearest
healthy service in a different datacenter!

66. Under the Hood: Network Tomography
• Rides on pings that are part of LAN and WAN
gossip
• Models networking round trip time using simple
physics simulation with masses and springs
• Develops a set of “network coordinates” for round
trip time estimation with a simple calculation

67. Under the Hood: Network Tomography

68. HASHICORP
Consul
Conclusion

69. HASHICORP
Key Value Store
HTTP API
Host & Service
Level Health
Checks
Datacenter Aware
Consul solves four central challenges with SOA
Service
Discovery
HTTP + DNS

70. HASHICORP
Further reading
- Consul vs. Other Software:
consul.io/intro/vs/index.html
- Consul Agent:
consul.io/docs/agent/basics.html
- Consul Commands:
consul.io/docs/commands/index.html
- Consul Internals:
consul.io/docs/internals/index.html

71. Questions?

72. Thanks!
sean@hashicorp.com