Why pay for always-on relational database service when you can deploy it yourself so easily? This demo-heavy talk will show off a deceptively simple high availability stack for PostgreSQL, using Docker, Etcd, Kubernetes, Patroni and Atomic. Not only is this open source solution ready to go to give you HA Postgres right now, it represents an approach which can be adapted to other relational databases with replication. Sched Link: http://sched.co/6BV4

1. Full
Auto
Database
Josh Berkus
Red Hat Project Atomic
KubeCon.EU 2016

4. WIP: waiting for
1.2/1.3

6. Demo

7. Single Master DBs:
Problem
low availability
unidirectional replication
very manual HA solutions

8. Why not multi­master
DBs?
just moving the problem around
"eventual" consistency
network lag
maturity issues
feature poverty
app compatibility

9. But PG Replicaton is
Awesome!
Easy to set up
Guaranteed
Corruption­free
Anti­footgun
Combines with DR

10. Y U No Failover?

11. "Automated
failover is too
complicated.
You don't want
it."

12. NO!

13. Hard != Impossible

14. Hard != Impossible
general autofailover is prohibitive
but ... we can implement common
use cases

15. The 80% Solution
1. Pool of async replicas
2. Cheap/replacable nodes
Containers
3. Watchdog service
4. Auto­promote one replica
5. Other nodes remaster
6. Update routing

16. Now, a little history ...

17. Handyrep
master­controller architecture
based on Python Fabric + SSH
worked in production
worked with any Postgres config
pluggable
www.handyrep.org

18. Handyrep: too general
Difficult to install
Difficult to debug
Over 100 configuration options
Scaled poorly
HR server was SPoF

19. Zalando
no1 European online fashion
15m customers
150 databases
24/7/365 operation
... needed automated,
decentralized HA

20. False failover
Misfires
Race
conditions
Failover Failure

22. Split Brain and S­M
DBs
worst possible outcome
automated recovery impossible
manual recovery painful

23. Patroni

25. 1. Postgres is a poor store of its own
replication state
2. Smart agents > top­down
controllers

26. Compose Governor
Containers
Etcd­based consensus
Simple PostgreSQL controller
... so we forked it.

27. How it works

28. failover est omnis divisa in
partes tres

29. failover est omnis divisa in
partes tres

30. The Patroni Controller

31. Patroni controller
Python daemon
Runs in each container as PID 1
Controls Postgres
startup/shutdown/config
Provides external REST API
Enforces opinionated config

32. Patroni Failover

33. Patroni Failover

34. Patroni Failover

35. Patroni Failover

36. Patroni Failover

37. Patroni Failover

38. Patroni Failover

39. Patroni Failover

40. Patroni Failover

41. Patroni Failover

42. Patroni Failover

43. Patroni Failover

44. Patroni Failover

45. Patroni Failover

46. Patroni Failover

47. What about split­brain?

48. Etcd
distributed consensus HTTP data
store
Raft algoritm
implements CA
great for config + metadata
not for data data

49. Etcd Alternatives
Zookeeper
larger scale
supported
Consul
integrates discovery
not (yet) suppported

50. What's AtomicDB?
WIP project
PostgreSQL
Patroni
Atomic Host
Kubernetes
Dynamic proxy (dev)
Cockpit UI (dev)

51. Let's see that
again

52. The Proxy Problem
differentiate master and read­only
connections
master service needs to follow
failover
failover logic too complex for
kubernetes (1.1)

53. pgbouncer?
current implementation in
pgbouncer
master, read slaves separate
services/ports
depends on flannel LB
not good enough. Waiting for
1.2/1.3!

54. More features
pg_rewind support (9.4+)
configurable node imaging
WAL­E
PITR
synchronous replication
non­failover replicas

55. More Stuff Under
development
cascading replication
integrated proxy
BDR support?
fork us on Github!

56. Resources
This Presentation:
jberkus.github.io/full_auto_db
Patroni Project:
github.com/zalando/patroni
AtomicDB Project:
github.com/jberkus/atomicdb

57. more
jberkus:
project
atomic:
@fuzzychef
www.databasesoup.com
www.projectatomic.io
RedHat booth for
Cockpit Kube demo
¿questions?