DataOpsbarcelona 2019: Deep dive into MySQL Group Replication... the magic explained

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purpose only, and may not be
incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied up in
making purchasing decisions. The development, release and timing of any features or functionality described for Oracle´s product
remains at the sole discretion of Oracle.
Copyright @ 2019 Oracle and/or its affiliates. All rights reserved.
3 / 191

about.me/lefred
Who am I ?
4 / 191

Frédéric Descamps
@lefred
MySQL Evangelist
Managing MySQL since 3.23
devops believer
living in Belgium 🇧🇪
https://lefred.be

5 / 191

Group Replication: heart of MySQL InnoDB Cluster
6 / 191

Group Replication: heart of MySQL InnoDB Cluster
7 / 191

MySQL Group Replication
but what is it ?!?
8 / 191

GR is a plugin for MySQL, made by MySQL and packaged with
MySQL
GR is an implementation of Replicated Database State
Machine theory
Paxos based protocol
GR allows to write on all Group Members (cluster nodes)
simultaneously while retaining consistency
GR implements conflict detection and resolution
GR allows automatic distributed recovery
Automatic failover
Automatic membership configuration
Adding/removing members
Network partitions, failures
Prevents data loss
Supported on all MySQL platforms !!
Linux, Windows, Solaris, OSX, FreeBSD
Completely Open Source - GPL ! No license required to have
High Availability
but what is it ?!?
9 / 191

Group Replication is nice, but how does it work ?
10 / 191

it´s just ...
11 / 191

it´s just ...
12 / 191

it´s just ...
... no, in fact the writesets replication is synchronous and then certification and apply of the changes are local to each nodes and
asynchronous.
13 / 191

it´s just ...
... no, in fact the writesets replication is synchronous and then certification and apply of the changes are local to each nodes and
asynchronous.
not that easy to understand... right ? As a picture is worth a 1000 words, let´s illustrate this...
14 / 191

15 / 191

16 / 191

17 / 191

18 / 191

19 / 191

20 / 191

21 / 191

22 / 191

23 / 191

24 / 191

25 / 191

26 / 191

27 / 191

MySQL Group Replication (full transaction)
28 / 191

29 / 191

30 / 191

31 / 191

32 / 191

33 / 191

34 / 191

35 / 191

36 / 191

MySQL Group Communication System (GCS)
MySQL Xcom protocol
37 / 191

MySQL Xcom protocol
Replicated Database State Machine
38 / 191

MySQL Xcom protocol
Paxos based protocol (a variant of Mencius)
39 / 191

MySQL Xcom protocol
its task: deliver messages across the distributed system:
40 / 191

MySQL Xcom protocol
atomically
41 / 191

MySQL Xcom protocol
atomically
in Total Order (Writes)
42 / 191

MySQL Xcom protocol
atomically
in Total Order (Writes)
MySQL Group Replication receives the Ordered 'tickets' from this GCS subsystem.
43 / 191

Architecture, Stack, Core, ...
44 / 191

MySQL Group Replication - Architecture
Node Types:
R: Traffic routers/proxies: MySQL Router, HAProxy, ProxySQL...
M: mysqld nodes participating in MySQL Group Replication
45 / 191

MySQL Group Replication - Stack
46 / 191

Server calls into the plugin through a generic interface
Most server internals are hidden from the plugin
Plugin interacts with the server through a generic interface
Replication plugin determines the fate of the commit
operation through a well defined server interface
The plugin makes use of the relay log infrastructure to
inject changes in the receiving server
MySQL Group Replication - Core
47 / 191

Maintaining distributed execution context
Detecting and Resolving conflicts
Handling distributed recovery
Detect membership changes
Donate state if needed
Collect state if needed
Proposing transactions to other members
Receiving and handling transactions from other members
Deciding the ultimate fate of transactions
commit or rollback
MySQL Group Replication - GR Plugin
48 / 191

The communication API (and binding) is responsible for:
Abstracting the underlying group communication system implementation from
the plugin itself
Mapping the interface to a specific group communication system implementation
The Group Communication System engine:
Paxos implementation
(Similar to Paxos Mencius)
Building block to provide distributed agreement between servers
MySQL Group Replication - GCS
49 / 191

Total Order
GTID generation
50 / 191

MySQL Group Replication uses MySQL replication framework by
design:
binary logs
51 / 191

design:
binary logs
relay logs
52 / 191

design:
binary logs
relay logs
GTIDs: Global Transaction IDs
53 / 191

How does Group Replication handle GTIDs ?
There are two ways of generating GTIDs:
54 / 191

AUTOMATIC: the transaction is assigned with an automatically generated id during commit. Where regular replication uses the
source server UUID, on Group Replication, the group name is used.
55 / 191

AUTOMATIC: the transaction is assigned with an automatically generated id during commit. Where regular replication uses the
source server UUID, on Group Replication, the group name is used.
ASSIGNED: the user assigns manually a GTID through SET GTID_NEXT to the transaction. This is common to any replication
format and the id is assigned before the transaction starts.
56 / 191

57 / 191

Group Replication : Total Order Delivery - GTID
58 / 191

59 / 191

60 / 191

61 / 191

62 / 191

63 / 191

64 / 191

65 / 191

66 / 191

67 / 191

68 / 191

69 / 191

70 / 191

71 / 191

72 / 191

73 / 191

74 / 191

75 / 191

76 / 191

Group Replication : GTID
The previous example was not totally in sync with reality. In fact, a writer allocates a block of GTID and when we have multiple
writes (multi-primary mode) all writers will use GTID sequence numbers in their allocated block.
The size of the block is defined by group_replication_gtid_assignment_block_size (default to 1M)
77 / 191

Example:
Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355
78 / 191

Example:
New write on an other node:
Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355,1000354
79 / 191

Example:
Let's write on the third node:
Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355:1000354:2000354
80 / 191

Example:
Let's write on the third node:
And writing back on the first one:
81 / 191

82 / 191

done !
Return from Commit
83 / 191

Group Replication: return from commit
Asynchronous Replication:
84 / 191

Group Replication: return from commit (2)
Semi-Sync Replication:
85 / 191

(*): eventual
Group Replication (*):
86 / 191

(*): before
87 / 191

(*): after
88 / 191

Does this mean we can have a distant node and always let it ack later
?
89 / 191

?
NO!
90 / 191

?
NO!
Because the system has to wait for the noop (single skip message) from the “distant” node where latency is higher
The size of the GCS consensus messages window can be get and set from UDF functions:
group_replication_get_write_concurrency(), group_replication_set_write_concurrency()
mysql> select group_replication_get_write_concurrency();
+-------------------------------------------+
| group_replication_get_write_concurrency() |
+-------------------------------------------+
| 10 |
+-------------------------------------------+
91 / 191

Event Horizon
GCS Write Consensus Concurrency
92 / 191

Event Horizon
group replication write concurrency
93 / 191

Event Horizon
94 / 191

Event Horizon
95 / 191

Event Horizon
96 / 191

Event Horizon
97 / 191

Event Horizon
98 / 191

Event Horizon
99 / 191

100 / 191

conflict
Optimistic Locking
101 / 191

Group Replication : Optimistic Locking
Group Replication uses optimistic locking
102 / 191

during a transaction, local (InnoDB) locking happens
103 / 191

optimistically assumes there will be no conflicts across nodes
(no communication between nodes necessary)
104 / 191

cluster-wide conflict resolution happens only at COMMIT, during certification
105 / 191

cluster-wide conflict resolution happens only at COMMIT, during certification
Let´s first have a look at the traditional locking to compare.
106 / 191

Traditional locking
107 / 191

Traditional locking
108 / 191

Traditional locking
109 / 191

Traditional locking
110 / 191

Traditional locking
111 / 191

Traditional locking
112 / 191

Optimistic Locking
113 / 191

Optimistic Locking
114 / 191

Optimistic Locking
115 / 191

Optimistic Locking
116 / 191

Optimistic Locking
117 / 191

Optimistic Locking
118 / 191

Optimistic Locking
The system returns error 149 as certification failed:
ERROR 1180 (HY000): Got error 149 during COMMIT
119 / 191

Optimistic Locking (2)
The conflict detection can happen at two different stage:
Only one of the two transactions was sent to the Group and the other one is still running (local).
If both transaction were sent to the Group at almost the same time and both reached GCS/XCOM.
120 / 191

Only one transaction was sent to GCS and the other conflicting one is still local:
In this case, it's the high priority transaction mechanism of InnoDB that kills the local one:
If you try any statement in the transaction's session you will see:
ERROR: 1213: Deadlock found when trying to get lock; try restarting transaction
If you try to commit the transaction you will see:
ERROR: 1180: Got error 149 - 'Lock deadlock; Retry transaction' during COMMIT
121 / 191

Both transactions were sent to GCS:
The second transaction (the conflicting one) will return:
ERROR 3101 (40000): Plugin instructed the server to rollback the current transaction.
122 / 191

Such conflicts happen only when using multi-primary group !

not totally true in MySQL < 8.0.13 when failover happens
123 / 191

Drawbacks of optimistic locking
having a first-committer-wins system means conflicts will more likely happen when writing on multiple members with:
124 / 191

large transactions
125 / 191

large transactions
long running transactions
126 / 191

large transactions
long running transactions
hotspot records
127 / 191

128 / 191

Configurable Consistency Guarantees
Consistency Levels
129 / 191

mysql> show variables like
'group_replication_consistency';
+-------------------------------+----------+
| Variable_name | Value |
+-------------------------------+----------+
| group_replication_consistency | EVENTUAL |
+-------------------------------+----------+

Consistency: EVENTUAL (default)
By default, there is no synchronization point for the transactions, when you perform a write on a node, if you immediately read the
same data on another node, it is eventually there.
130 / 191

mysql> show variables like
'group_replication_consistency';
+-------------------------------+----------+
+-------------------------------+----------+
| group_replication_consistency | EVENTUAL |
+-------------------------------+----------+

Consistency: EVENTUAL (default)
By default, there is no synchronization point for the transactions, when you perform a write on a node, if you immediately read the
same data on another node, it is eventually there.
Since MySQL 8.0.16, we have to possibility to set the
synchronization point at read or at write or both (globally or for a
session).
131 / 191

Consistency: BEFORE (READ)
132 / 191

Consistency: AFTER (WRITE)
133 / 191

Consistency: BEFORE_AND_AFTER
134 / 191

135 / 191

can the transaction be committed ?
Certification
136 / 191

Certification
Certification is the process that only needs to answer the following unique question:
137 / 191

Certification
can the write (transaction) be committed ?
138 / 191

Certification
based on yet to be applied transactions
139 / 191

Certification
such conflicts must come for other members/nodes
140 / 191

Certification
happens on every member/node and is deterministic
141 / 191

Certification
results are not reported to the group (does not require a new communication step)
142 / 191

Certification
pass: commit/queue to appy
143 / 191

Certification
fail: rollback/drop the transaction
144 / 191

Certification
serialized by the total order in GCS/XCOM + GTID
145 / 191

Certification
serialized by the total order in GCS/XCOM + GTID
cost is based on trx size (# rows & # keys)
146 / 191

terminology
Write vs Writeset
147 / 191

Let's illustrate a table:
148 / 191

Now let's make a change
149 / 191

and at commit time:
150 / 191

Writesets
Contain the hash for the rows PKs that are changed and in some cases the hashes of foreign keys or others dependencies that
need to be captured (e.g. non NULL UKs). Writesets are gathered during transaction execution.
151 / 191

Writesets
Contain the hash for the rows PKs that are changed and in some cases the hashes of foreign keys or others dependencies that
need to be captured (e.g. non NULL UKs). Writesets are gathered during transaction execution.
Writes
Called also write values, refers to the actual changes. Write values are also gathered during transaction execution.
152 / 191

Writeset - examples
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
mysql> insert into t2 values (1,2);
154 / 191

Writeset - examples
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
pke: PRIMARY | test | t2 | 1 | 1 hash: 11853456929268668462
155 / 191

Writeset - examples
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
mysql> update t2 set name=3 where id=1;
156 / 191

Writeset - examples
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
157 / 191

+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
mysql> insert into t3 values (1,2,3);
159 / 191

+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
161 / 191

+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
pke: name | test | t3 | 3 hash: 18082071075512932388
162 / 191

+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
+-------+-----------+------+-----+---------+-------+
[after image]
[before image]
163 / 191

Certification
164 / 191

165 / 191

Enhanced support for large transactions
Message Fragmentation
166 / 191

Fragmenting an outgoing message
(1). If the message size exceeds the maximum size that the user allows
(group_replication_communication_max_message_size), the member fragments the message into chunks
that do not exceed the maximum size.
(2). The member broadcasts each chunk to the group, i.e. forwards each chunk individually to XCom.
167 / 191

Reassembling an incoming message - first fragment
(2). The members conclude that the incoming message is actually a fragment of a bigger message.
(3). The members buffer the incoming fragment because they conclude the fragment is a chunk of a still-incomplete message.
(Fragments contain the necessary metadata to reach this conclusion.)
168 / 191

Reassembling an incoming message - second fragment
(5). Same as step 3.
(6). Same as step 4.
169 / 191

Reassembling an incoming message - last fragment
1. The members conclude that the incoming message is actually a fragment of a bigger message.
2. The members conclude that the incoming fragment is the last chunk missing, reassemble the original, whole message, and
process it.
170 / 191

Houston we have a problem !
Flow Control
171 / 191

Flow Control
In Group Replication, every member send statistics about its queues (applier queue and certification queue) to the other members.
Then every node decide to slow down or not if they realize that one node reached the threshold for one of the queue.
172 / 191

Flow Control
So when group_replication_ ow_control_mode is set to QUOTA on the node seeing that one of the other
members of the cluster is lagging behind (threshold reached), it will throttle the write operations to the a quota that is calculated
based on the number of transactions applied in the last second, and then it is reduced below that by subtracting the “over the
quota” messages from the last period.
173 / 191

Flow Control
So when group_replication_ ow_control_mode is set to QUOTA on the node seeing that one of the other
members of the cluster is lagging behind (threshold reached), it will throttle the write operations to the a quota that is calculated
based on the number of transactions applied in the last second, and then it is reduced below that by subtracting the “over the
quota” messages from the last period.
This mean that the threshold is NOT decided on the node being slow, but the node writing a transaction checks its threshold flow
control values and compare them to the statistics from the other nodes to decide to throttle or not.
174 / 191

Flow Control - on writer
>quota
175 / 191

Flow Control - on all members
176 / 191

Flow Control - configuration variables
As in MySQL 8.0.16:
+-----------------------------------------------------+-------+
+-----------------------------------------------------+-------+
| group_replication_ ow_control_applier_threshold | 25000 |
| group_replication_ ow_control_certi er_threshold | 25000 |
| group_replication_ ow_control_hold_percent | 10 |
| group_replication_ ow_control_max_quota | 0 |
| group_replication_ ow_control_member_quota_percent | 0 |
| group_replication_ ow_control_min_quota | 0 |
| group_replication_ ow_control_min_recovery_quota | 0 |
| group_replication_ ow_control_mode | QUOTA |
| group_replication_ ow_control_period | 1 |
| group_replication_ ow_control_release_percent | 50 |
+-----------------------------------------------------+-------+
177 / 191

transaction's lifecycle in Group Replication
Summary
178 / 191

begin;
179 / 191

begin;
update table1
set c = 999
where id =2;
180 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
181 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
clientblocksoncommit...
182 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
writesets
+ gtid_event
+ write values
183 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
writesets
+ gtid_event
+ write values
certify
184 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
writesets
+ gtid_event
+ write values
certify
certify
185 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
writesets
+ gtid_event
+ write values
certify
certify
certify
186 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
commit finalized
writesets
+ gtid_event
+ write values
certify
certify
certify
+ GTID
bin
log
187 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
commit finalized
writesets
+ gtid_event
+ write values
certify
certify
certify
+ GTID
bin
log
+ GTID
188 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
commit finalized
writesets
+ gtid_event
+ write values
certify
certify
certify
+ GTID
bin
log
+ GTID
+ GTIDrelay
log
relay
log
189 / 191

begin;
update table1
set c = 999
where id =2;
update table1
set b = "eee"
where id = 3;
commit;
commit finalized
writesets
+ gtid_event
+ write values
certify
certify
certify
+ GTID
bin
log
+ GTID
+ GTIDrelay
log
relay
log
bin
log
bin
log
190 / 191

Thank you !
Any Questions ?
191 / 191

DataOpsbarcelona 2019: Deep dive into MySQL Group Replication... the magic explained

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to DataOpsbarcelona 2019: Deep dive into MySQL Group Replication... the magic explained

Similar to DataOpsbarcelona 2019: Deep dive into MySQL Group Replication... the magic explained (20)

More from Frederic Descamps

More from Frederic Descamps (20)

Recently uploaded

Recently uploaded (20)

DataOpsbarcelona 2019: Deep dive into MySQL Group Replication... the magic explained