Espresso Database Replication with Kafka, Tom Quiggle

Distributed Data Systems 1©2016 LinkedIn Corporation. All Rights Reserved.
ESPRESSO Database Replication with Kafka
Tom Quiggle
Principal Staff Software Engineer
tquiggle@linkedin.com
www.linkedin.com/in/tquiggle
@TomQuiggle

 ESPRESSO Overview
– Architecture
– GTIDs and SCNs
– Per-instance replication (0.8)
– Per-partition replication (1.0)
 Kafka Per-Partition Replication
– Requirements
– Kafka Configuration
– Message Protocol
– Producer
– Consumer
 Q&A
Agenda

ESPRESSO Overview

 Hosted, Scalable, Data as a Service (DaaS) for LinkedIn’s Online
Structured Data Needs
 Databases are partitioned
 Partitions distributed across available hardware
 HTTP proxy routes requests to appropriate database node
 Apache Helix provides centralized cluster management
ESPRESSO1
1. Elastic, Scalable, Performant, Reliable, Extensible, Stable, Speedy and Operational

ESPRESSO Architecture
Storage Node
API Server
MySQL
Router
Router
Router
Apache Helix
ZooKeeper
Storage Node
API Server
MySQL
Storage Node
API Server
MySQL
Storage Node
API Server
MySQL
Data
Control
Routing Table
r
r
r
HTTP
Client
HTTP

GTIDs and SCNs
MySQL 5.6 Global Transaction Identifier
 Unique, monotonically increasing, identifier for each
transaction committed
 GTID :== source_id:transaction_id
 ESPRESSO conventions
– source_id encodes database name and partition number
– transaction_id is a 64 bit numeric value
 High Order 32 bits is generation count
 Low order 32 bit are sequence within generation
– Generation increments with every change in mastership
– Sequence increases with each transaction
– We refer to a transaction_id component as a Sequence Commit Number (SCN)

GTIDs and SCNs
Example binlog transaction:
SET @@SESSION.GTID_NEXT= 'hash(db_part):(gen<<32 + seq)';
SET TIMESTAMP=<seconds_since_Unix_epoch>
BEGIN
Table_map: `db_part`.`table1` mapped to number 1234
Update_rows: table id 1234
BINLOG '...'
BINLOG '...'
Table_map: `db_part`.`table2` mapped to number 5678
Update_rows: table id 5678
BINLOG '...'
COMMIT

Node 1
P1 P2 P3
Node 2
P1 P2 P3
Node 3
P1 P2 P3
Node 1
P4 P5 P6
Node 2
P4 P5 P6
Node 3
P4 P5 P6
ESPRESSO: 0.8 Per-Instance Replication
Master
Slave
Offline

Node 1
P1 P2 P3
Node 2
P1 P2 P3
Node 3
P1 P2 P3
Node 1
P4 P5 P6
Node 2
P4 P5 P6
Node 3
P4 P5 P6
Master
Slave
Offline

Issues with Per-Instance Replication
 Poor resource utilization – only 1/3 of nodes service application requests
 Partitions unnecessarily share fate
 Cluster expansion is an arduous process
 Upon node failure, 100% of the traffic is redirected to one node

ESPRESSO: 1.0 Per-Partition Replication
Per-Instance MySQL replication replaced with Per-Partition Kafka
HELIX
P4:
Master: 1
Slave: 3
…
EXTERNALVIEW
Node 1
Node 2
Node 3
LIVEINSTANCES
Node 1
P1 P2
P4
P3
P5 P6
P9 P10
Node 2
P5 P6
P8
P7
P1 P2
P11 P12
Node 3
P9 P10
P12
P11
P3 P4
P7 P8
Kafka

Cluster Expansion
Initial State with 12 partitions, 3 storage nodes, r=2
HELIX
EXTERNALVIEW
Node 1
Node 2
Node 3
LIVEINSTANCES
Node 1
P1 P2
P4
P3
P5 P6
P9 P10
Node 2
P5 P6
P8
P7
P1 P2
P11 P12
Node 3
P9 P10
P12
P11
P3 P4
P7 P8
Master
Slave
Offline
P4:
Master: 1
Slave: 3
…

Cluster Expansion
Adding Node: Helix Sends OfflineToSlave for new partitions
HELIX
EXTERNALVIEW
Node 1
Node 2
Node 3
Node 4
LIVEINSTANCES
Node 1
P1 P2
P4
P3
P5 P6
P9 P10
Node 2
P5 P6
P8
P7
P1 P2
P11 P12
Node 3
P9 P10
P12
P11
P3 P4
P7 P8
Node 4
P4 P8
P1
P12
P7 P9
P4:
Master: 1
Slave: 3
Offline: 4
…

Cluster Expansion
Once a new partition is ready, transfer ownership and drop old
HELIX
EXTERNALVIEW
Node 1
Node 2
Node 3
Node 4
LIVEINSTANCES
Node 1
P1 P2 P3
P5 P6
P9 P10
Node 2
P5 P6
P8
P7
P1 P2
P11 P12
Node 3
P9 P10
P12
P11
P3 P4
P7 P8
Node 4
P4 P8
P1
P12
P7 P9
P4:
Master: 4
Slave: 3
…

Cluster Expansion
Continue migration of master and slave partitions
HELIX
EXTERNALVIEW
Node 1
Node 2
Node 3
Node 4
LIVEINSTANCES
Node 1
P1 P2 P3
P5 P6
P9 P10
Node 2
P5 P6 P7
P2
P11 P12
Node 3
P9 P10
P12
P11
P3 P4
P7 P8
Node 4
P4 P8
P1
P12
P7 P9
P9:
Master: 3
Slave: 1
Offline: 4
…

Cluster Expansion
Rebalancing is complete after last partition migration
HELIX
EXTERNALVIEW
Node 1
Node 2
Node 3
Node 4
LIVEINSTANCES
Node 1 Node 2
Node 3 Node 4
P4 P8
P1
P12
P7 P9
P5 P6
P2
P7
P11 P12
P9 P10
P3
P11
P4 P8
P1 P2
P5
P3
P6 P10
P9:
Master: 4
Slave: 3
…

Node Failover
During failure or planned maintenance, promote slaves to master
HELIX
EXTERNALVIEW
Node 1
Node 2
Node 3
Node 4
LIVEINSTANCES
Node 1 Node 2
Node 3 Node 4
P4 P8
P1
P12
P7 P9
P5 P6
P2
P7
P11 P12
P9 P10
P3
P11
P4 P8
P1 P2
P5
P3
P6 P10
P9:
Master: 4
Slave: 3
…

Node Failover
During failure or planned maintenance, promote slaves to master
HELIX
EXTERNALVIEW
Node 1
Node 2
Node 4
LIVEINSTANCES
Node 1 Node 2
Node 3 Node 4
P4 P8
P1
P12
P7 P9
P5 P6
P2
P7
P11 P12
P9 P10
P3
P11
P4 P8
P1 P2
P5
P3
P6 P10
P9:
Master: 4
Offline: 3
…

Advantages of Per-Partition Replication
 Better hardware utilization
– All nodes service application requests
 Mastership hand-off done in parallel
 After node failure, can restore full replication factor in parallel
 Cluster expansion is as easy as:
– Add node(s) to cluster
– Rebalance
 Single platform for all Change Data Capture
– Internal replication
– Cross-colo replication
– Application CDC consumers

Kafka Per-Partition Replication

Kafka for Internal Replication
Storage Node
MySQL
Open Replicator
Kafka Producer
API Server
binlog
binlog
event
Kafka Consumer
SQL
INSERT..UPDATE
SQL
INSERT..UPDATE
Storage Node
MySQL
Open Replicator
Kafka Producer
API Server
binlog
binlog
event
Kafka Consumer
SQL
INSERT..UPDATE
SQL
INSERT..UPDATE
Kafka Partition
Kafka Message
Kafka Message
Client
HTTP
PUT/POST

Requirements
Delivery Must Be:
 Guaranteed
 In-Order
 Exactly Once (sort of)

Broker Configuration
 Replication factor = 3
(most LinkedIn clusters use 2)
 min.isr=2
 Disable unclean leader elections

B – Begin txn
E – End txn
C – Control
Message Protocol
Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104
DB_0:
3:104
E

Message Protocol – Mastership Handoff
Old Master
MySQ
L
ProducerConsumer
Promoted Slave
MySQ
L
ProducerConsumer
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104
DB_0:
3:104
E
4:0
C

Master
MySQ
L
ProducerConsumer
Promoted Slave
MySQ
L
ProducerConsumer
Consumed
own control
message
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104
DB_0:
3:104
E
4:0
C

Old Master
MySQ
L
ProducerConsumer
Master
MySQ
L
ProducerConsumer
Enable
writes with
new gen
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104
DB_0:
3:104
E
4:0
C
4:0
B

Kafka Producer Configuration
 acks = “all”
 retries = Integer.MAX_VALUE
 block.on.buffer.full=true
 max.in.flight.requests.per.connection=1
 linger=0
 On non-retryable exception:
– destroy producer
– create new producer
– resume from last checkpoint

Kafka Producer Checkpointing
Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104
Can’t
Checkpoint
Here
Periodically writes (SCN, Kafka Offset) to MySQL table
May only checkpoint offset at end of valid transaction!

Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104
Producer checkpoint will lag current producer Kafka Offset
Kafka Offset obtained from callback
Last
Checkpoint
Here

Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
Last
Checkpoint
Here
send()
FAILS
X
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104 3:104

Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
Recreate producer and resume from last checkpoint
Resume
From
Checkpoint
Messages will be replayed
3:102
B
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104

Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
Kafka stream now contains replayed transactions
(possibly including partial transactions)
Can
Checkpoint
Here
Replayed Messages
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104 3:102
B
3:102 3:102
E
3:103
B,E
3:104
B
3:104

Partition 3
Kafka Consumer
 Uses Low Level Consumer
 Consume Kafka partitions slaved on node
Partition 1
Partition 2
Kafka Broker A
Kafka Broker B
Kafka Consumer
poll()
Consumer Thread
EspressoKafkaConsumer
EspressoReplicationApplier
MySQL
P1
P2
P3
Applier
Threads

Kafka Consumer
Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104
Slave updates (SCN, Kafka Offset) row for every committed txn
3:101@2

Kafka Consumer
Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
Client only applies messages with SCN greater than last committed
Replayed Messages
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104 3:102
B
3:102 3:102
E
3:103
B,E
3:104
B
3:104
BEGIN
Transaction
3:104
3:103@6

Kafka Consumer
Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
Incomplete transaction is rolled back
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104 3:102
B
3:102 3:102
E
3:103
B,E
3:104
B
3:104
ROLLBACK
3:104
Replayed Messages
3:103@6

Kafka Consumer
Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
Client only applies messages with SCN greater than last committed
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104 3:102
B
3:102 3:102
E
3:104
B
3:104
SKIP
3:102..3:10
3
Replayed Messages
3:103@6
3:103
B,E

Kafka Consumer
Master
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
3:101
B,E
3:102
B
3:102 3:102
E
3:100
B,E
3:103
B,E
3:104
B
3:104 3:102
B
3:102 3:102
E
3:104
B
3:104
Replayed Messages
BEGIN
3:104
(again)
3:104
E
3:103@6
3:103
B,E

Zombie Write Filtering
 What if stalled master continues writing after transition?

MASTER
MySQ
L
ProducerConsumer
Slave
MySQ
L
ProducerConsumer
3:102
B
3:102 3:102
E
3:103
B,E
3:104
B
3:104
Master Stalled

Master
MySQ
L
ProducerConsumer
Promoted Slave
MySQ
L
ProducerConsumer
3:102
B
3:102 3:102
E
3:103
B,E
3:104
B
3:104
Master Stalled
4:0
C
Helix sends SlaveToMaster transition to one of the slaves

Master
MySQ
L
ProducerConsumer
New Master
MySQ
L
ProducerConsumer
Master Stalled
3:102
B
3:102 3:102
E
3:103
B,E
3:104
B
3:104 4:0
C
4:1
B,E
4:2
B
4:2
E
Slave becomes master and starts taking writes

Master
MySQ
L
ProducerConsumer
New Master
MySQ
L
ProducerConsumer
3:102
B
3:102 3:102
E
3:103
B,E
3:104
B
3:104 4:0
C
4:1
B,E
4:2
B
4:2
E
3:104
E
3:105
B,E
Stalled Master resumes and sends binlog entries to Kafka

ERROR
MySQ
L
ProducerConsumer
New Master
MySQ
L
ProducerConsumer
3:102
B
3:102 3:102
E
3:103
B,E
3:104
B
3:104 4:0
C
4:1
B,E
4:2
B
4:2
E
3:104
E
3:105
B,E
4:3
B,E
Former master goes into ERROR state
Zombie writes filtered by all consumers based on increasing SCN rule

Current Status

ESPRESSO Kafka Replication: Current Status
 Pre-Production integration environment migrated to Kafka replication
 8 production clusters migrated (as of 4/11)
 Migration will continue through Q3 of 2016
 Average replication latency < 90ms

Conclusions
 Configure Kafka for reliable, at least once, delivery. See:
http://www.slideshare.net/JiangjieQin/no-data-loss-pipeline-with-apache-kafka-49753844
 Carefully control producer and consumer checkpoints along txn boundaries
 Embed sequence information in message stream to implement exactly-
once application of messages

Distributed Data Systems
Even our workspace is
Horizontally Scalable!

Espresso Database Replication with Kafka, Tom Quiggle

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Espresso Database Replication with Kafka, Tom Quiggle

Similar to Espresso Database Replication with Kafka, Tom Quiggle (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Espresso Database Replication with Kafka, Tom Quiggle

Editor's Notes