Percona XtraDB Cluster (PXC) is a 100% synchronized cluster in regards to DML operations. It is ensured by the optimistic locking model and ability to rollback transaction which cannot be applied on all nodes. However, DDL operations are not transactional in MySQL. This adds complexity when you need to change the schema of the database. Changes made by DDL may affect the results of the queries. Therefore all modifications must replicate on all nodes prior to the next data access. For operations that run momentarily, it can be easily achieved, but schema changes may take hours to apply. Therefore in addition to the safest synchronous blocking schema upgrade method: TOI, - PXC supports more relaxed, though not safe, method RSU. RSU: Rolling Schema Upgrade is advertised to be non-blocking. But you still need to take care of updates, running while you are performing such an upgrade. Surprisingly, even updates on not related tables and schema can cause RSU operation to fail. In this talk, I will uncover nuances of PXC schema upgrades and point to details you need to take special care about. Further Information Schema change is a frequent task, and many do not expect any surprises with it. However, the necessity to replay the changes to all synchronized nodes adds complexity. I made a webinar on a similar topic which was recorded and available for replay. Now I have found that I share a link to the webinar to my Support customers approximately once per week. Not having a good understanding of how schema change works in the cluster leads to lockups and operation failures. This talk will provide a checklist that will help to choose the best schema change method. Presented at Percona Live Online: https://perconaliveonline2020.sched.com/event/ePm2/how-to-avoid-pitfalls-in-schema-upgrade-with-percona-xtradb-cluster

1. How to Avoid Pitfalls in Schema Upgrade
with Percona XtraDB Cluster
Sveta Smirnova
Percona

2. Sveta Smirnova
• MySQL Support engineer
• Author of
• MySQL Troubleshooting
• JSON UDF functions
• FILTER clause for MySQL
• Speaker
• Percona Live, OOW, Fosdem,
DevConf, HighLoad... 2

3. Table of Contents
•TOI
•RSU
•pt-online-schema-change (pt-osc)
3

4. Introduction
• Galera Replication Library
• Provides synchronous replication for MySQL
4

5. Introduction
• Galera Replication Library
• Provides synchronous replication for MySQL
• Galera Clusters
Percona XtraDB Cluster
MariaDB Galera Cluster
Galera Cluster for MySQL
4

6. How Galera works
• Data modification happens on a node
• Optimistic locking control
5

7. How Galera works
• Data modification happens on a node
• Optimistic locking control
• While a transaction is in progress
• Writes are applied locally
• Same as for standalone server
5

8. How Galera works
• Data modification happens on a node
• Optimistic locking control
• While a transaction is in progress
• At the COMMIT time
• Broadcasts write set for the cluster
• Waits confirmation of the successful update
From all other nodes
Yes Commits transaction locally
No Rollbacks transaction
5

9. Data Updates
• Committed on all nodes or nowhere
• Safe
6

10. Challenges of DDL
• Replicated independently from storage engine
7

11. Challenges of DDL
• Replicated independently from storage engine
• Changes may affect query results
• Adding/removal of UNIQUE keys
• Adding/removal columns
• Changing column definition
7

12. Challenges of DDL
• Replicated independently from storage engine
• Changes may affect query results
• Modification can happen on any node
• The schema must be upgraded before DML
• There is no way to rollback schema upgrade
• MDLs are set only on one node
Not across the cluster
Not possible to rely on them for all nodes
Additional control required
7

13. TOI

14. Total Order Isolation (TOI)
• DDL changes are replicated in the same order
• Regarding other transactions
• All nodes are in the absolutely same state
• At any point of time
9

15. TOI: Illustration
• 3-nodes cluster
• Node A
• Node B
• Node C
10

16. TOI: Illustration
• Initial state
Node A
INSERT(103)
UPDATE(104)
ALTER(105)
Node B
UPDATE(101)
INSERT(102)
DELETE(108)
UPDATE(109)
Node C
SELECT(100)
INSERT(112)
SELECT(113)
UPDATE(114)
10

17. TOI: Illustration
• Queries status
Node A
INSERT(103)
UPDATE(104)
ALTER(105)
Node B
UPDATE(101)
INSERT(102)
DELETE(108)
UPDATE(109)
Node C
SELECT(100)
INSERT(112)
SELECT(113)
UPDATE(114)
10

18. TOI: Illustration
• ALTER in progress
Node A
ALTER(105)
Node B
DELETE(108)
UPDATE(109)
Node C
INSERT(112)
SELECT(113)
UPDATE(114)
10

19. TOI: Illustration
• ALTER finished
Node A Node B
DELETE(108)
UPDATE(109)
Node C
INSERT(112)
SELECT(113)
UPDATE(114)
10

20. PROCESSLIST: DML before ALTER
DML node> select DB, COMMAND, TIME, STATE, INFO from information_schema.processlist WHERE DB=’sbtest’;
+--------+---------+------+---------------------------------------------------+-----------------------+
| DB | COMMAND | TIME | STATE | INFO |
+--------+---------+------+---------------------------------------------------+-----------------------+
| sbtest | Query | 1 | wsrep: initiating pre-commit for write set (2886) | COMMIT |
| sbtest | Query | 1 | wsrep: initiating pre-commit for write set (2888) | COMMIT |
| sbtest | Query | 1 | wsrep: initiating pre-commit for write set (2884) | COMMIT |
| sbtest | Query | 1 | updating | DELETE FROM sbtest1.. |
| sbtest | Query | 1 | wsrep: initiating pre-commit for write set (2887) | COMMIT |
| sbtest | Query | 0 | wsrep: initiating pre-commit for write set (2889) | COMMIT |
| sbtest | Query | 1 | wsrep: initiating pre-commit for write set (2885) | COMMIT |
| sbtest | Query | 1 | wsrep: pre-commit/certification passed (2883) | COMMIT |
+--------+---------+------+---------------------------------------------------+-----------------------+
8 rows in set (0.00 sec)
11

21. PROCESSLIST: SELECT before ALTER
SELECT node> select DB, COMMAND, TIME, STATE, INFO from information_schema.processlist
-> WHERE DB=’sbtest’;
+--------+---------+------+-------------------+---------------------------------------+
| DB | COMMAND | TIME | STATE | INFO |
+--------+---------+------+-------------------+---------------------------------------+
| sbtest | Query | 0 | statistics | SELECT pad FROM sbtest2 WHERE id=5009 |
| sbtest | Query | 0 | starting | SELECT pad FROM sbtest3 WHERE id=4951 |
| sbtest | Query | 0 | statistics | SELECT pad FROM sbtest4 WHERE id=4954 |
| sbtest | Query | 0 | System lock | SELECT pad FROM sbtest2 WHERE id=5351 |
| sbtest | Query | 0 | cleaning up | SELECT pad FROM sbtest2 WHERE id=4954 |
| sbtest | Sleep | 0 | | NULL |
| sbtest | Query | 0 | Sending to client | SELECT pad FROM sbtest1 WHERE id=4272 |
| sbtest | Query | 0 | closing tables | SELECT pad FROM sbtest4 WHERE id=4722 |
+--------+---------+------+-------------------+---------------------------------------+
8 rows in set (0.00 sec)
12

22. ALTER
DDL node> use ddltest;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
DDL node> alter table sbtest1 add key k1(c, k, pad);
Query OK, 0 rows affected (3 min 53.73 sec)
Records: 0 Duplicates: 0 Warnings: 0
13

23. PROCESSLIST: DML during ALTER
DML node> select DB, COMMAND, TIME, STATE, INFO from information_schema.processlist
-> WHERE DB in (’sbtest’,’ddltest’);
+---------+---------+------+---------------------------------------------------+----------------------+
| DB | COMMAND | TIME | STATE | |
+---------+---------+------+---------------------------------------------------+----------------------+
| sbtest | Query | 36 | wsrep: initiating pre-commit for write set (7886) | COMMIT |
| sbtest | Query | 37 | wsrep: initiating pre-commit for write set (7882) | COMMIT |
| sbtest | Query | 27 | wsrep: initiating pre-commit for write set (7887) | COMMIT |
| sbtest | Query | 27 | wsrep: initiating pre-commit for write set (7888) | COMMIT |
| sbtest | Query | 36 | wsrep: initiating pre-commit for write set (7885) | COMMIT |
| sbtest | Query | 37 | wsrep: initiating pre-commit for write set (7883) | COMMIT |
| sbtest | Query | 37 | wsrep: initiating pre-commit for write set (7884) | COMMIT |
| sbtest | Query | 10 | wsrep: initiating pre-commit for write set (7889) | COMMIT |
| ddltest | Sleep | 38 | altering table | alter table sbtest1. |
+---------+---------+------+---------------------------------------------------+----------------------+
9 rows in set (0.00 sec)
14

24. PROCESSLIST: SELECT during ALTER
SELECT node> select DB, COMMAND, TIME, STATE, INFO from information_schema.processlist
-> WHERE DB in (’sbtest’,’ddltest’);
+---------+---------+------+-------------------+-------------------------------------------+
| DB | COMMAND | TIME | STATE | |
+---------+---------+------+-------------------+-------------------------------------------+
| sbtest | Sleep | 0 | | NULL |
| sbtest | Sleep | 0 | | NULL |
| sbtest | Query | 0 | Sending to client | SELECT pad FROM sbtest4 WHERE id=4989 |
| sbtest | Sleep | 0 | | NULL |
| sbtest | Query | 0 | query end | SELECT pad FROM sbtest2 WHERE id=4961 |
| sbtest | Sleep | 0 | | NULL |
| sbtest | Sleep | 0 | | NULL |
| sbtest | Sleep | 0 | | NULL |
| ddltest | Sleep | 39 | altering table | alter table sbtest1 add key k1(c, k, pad) |
+---------+---------+------+-------------------+-------------------------------------------+
9 rows in set (0.14 sec)
15

25. TOI Advantages
• Data always consistent
• DDL applied to all nodes at the same time
• No failure due to schema inconsistency
16

26. TOI Disadvantages
• The whole cluster blocked
• For the duration of the entire DDL operation
• Schema upgrades replicated as a statement
• There is no guarantee that the ALTER succeed!
17

27. How to Perform Upgrade with TOI
• Schedule maintenance window
• Run DDL
• Cluster won’t be accessible until DDL ends
• SELECTs can continue
• wsrep sync wait != 1
18

28. When to Use TOI
• Quick DDL operations
19

29. When to Use TOI
• Quick DDL operations
• Creating new database objects
• CREATE DATABASE
• CREATE TABLE
19

30. When to Use TOI
• Quick DDL operations
• Creating new database objects
• Online operations which modify metadata only
• RENAME INDEX
• RENAME TABLE
• DROP INDEX
• ALGORITHM=INSTANT
Full list
19

31. RSU

32. Rolling Schema Upgrade (RSU)
• Variable wsrep OSU method
• Puts node into de-sync state
• For the duration of DDL
• Pauses Galera provider
• Schema can get out of sync!
21

33. User Responsibility
• Run DDL on the each node of the cluster
• Block read-write access that depend on DDL
• Until all nodes are in sync
• Make sure no write is performed to the table
• Until upgrade finishes on all nodes
• Failure makes cluster unrecoverable!
22

34. RSU Workflow
• User Action
• SET SESSION wsrep OSU method = ’RSU’;
• DDL
• Any other statement
• Node Operation
• Nothing
• Is wsrep OSU method set to RSU?
Yes Performs DDL
• Nothing
23

35. How Node Internally Executes DDL in RSU Mode
Does node have transactions in COMMIT mode?
24

36. How Node Internally Executes DDL in RSU Mode
Does node have transactions in COMMIT mode?
Yes Wait for 5 milliseconds
• wsrep RSU commit timeout (PXC only)
24

37. How Node Internally Executes DDL in RSU Mode
Does node have transactions in COMMIT mode?
Yes Wait for 5 milliseconds
• wsrep RSU commit timeout (PXC only)
Do transactions in the COMMIT mode still exist?
24

38. How Node Internally Executes DDL in RSU Mode
Does node have transactions in COMMIT mode?
Yes Wait for 5 milliseconds
• wsrep RSU commit timeout (PXC only)
Do transactions in the COMMIT mode still exist?
Yes Abort DDL
24

39. How Node Internally Executes DDL in RSU Mode
Does node have transactions in COMMIT mode?
No Put node into de-sync state
24

40. How Node Internally Executes DDL in RSU Mode
Does node have transactions in COMMIT mode?
No Put node into de-sync state
Pause write-set application
24

41. How Node Internally Executes DDL in RSU Mode
Does node have transactions in COMMIT mode?
No Put node into de-sync state
Pause write-set application
Execute DDL
24

42. How Node Internally Executes DDL in RSU Mode
Does node have transactions in COMMIT mode?
No Put node into de-sync state
Pause write-set application
Execute DDL
Bring the node back to the cluster
24

43. How Node Internally Executes DDL in RSU Mode
Does node have transactions in COMMIT mode?
No Put node into de-sync state
Pause write-set application
Execute DDL
Bring the node back to the cluster
Synchronize
24

44. RSU: Locking
• Not avoidable
• Updates to all objects on the node in RSU mode
must finish before the operation
• Failure aborts DDL
25

45. RSU Advantages
• Cluster remains functional
• Schedule long-running ALTER
• In the best time possible
26

46. RSU Disadvantages
• No checks for data and schema consistency
• This is your responsibility!
27

47. RSU Disadvantages
• No checks for data and schema consistency
• All writes must be stopped on the affected node
• Otherwise DDL fails with an error
27

48. RSU Disadvantages
• No checks for data and schema consistency
• All writes must be stopped on the affected node
• gcache should be big enough to hold changes
• Made while DDL was running
• Failure will cause SST when node re-joins cluster
• All schema changes will be lost
27

49. RSU Disadvantages
• No checks for data and schema consistency
• All writes must be stopped on the affected node
• gcache should be big enough to hold changes
• Any error can make cluster dysfunctional
27

50. RSU Disadvantages
• No checks for data and schema consistency
• All writes must be stopped on the affected node
• gcache should be big enough to hold changes
• Any error can make cluster dysfunctional
• Affected table must be offline
• Until the schema upgrade is done on all nodes
• Unless this is schema-compatible change
27

51. How to Use RSU
• Make sure gcache is big enough
• Must hold all updates while DDL is in progress
28

52. How to Use RSU
• Make sure gcache is big enough
• Must hold all updates while DDL is in progress
• Block all writes to the table/schema
28

53. How to Use RSU
Choose an ”upgrading node”
28

54. How to Use RSU
Choose an ”upgrading node”
Block all write requests to this node
28

55. How to Use RSU
Choose an ”upgrading node”
Block all write requests to this node
SET SESSION wsrep OSU method = ’RSU’;
28

56. How to Use RSU
Choose an ”upgrading node”
Block all write requests to this node
SET SESSION wsrep OSU method = ’RSU’;
Perform DDL in the same session
28

57. How to Use RSU
Choose an ”upgrading node”
Block all write requests to this node
SET SESSION wsrep OSU method = ’RSU’;
Perform DDL in the same session
SET SESSION wsrep OSU method = ’TOI’;
28

58. How to Use RSU
Choose an ”upgrading node”
Block all write requests to this node
SET SESSION wsrep OSU method = ’RSU’;
Perform DDL in the same session
SET SESSION wsrep OSU method = ’TOI’;
Re-enable writes
28

59. How to Use RSU
Choose an ”upgrading node”
Block all write requests to this node
SET SESSION wsrep OSU method = ’RSU’;
Perform DDL in the same session
SET SESSION wsrep OSU method = ’TOI’;
Re-enable writes
Repeat for other nodes
28

60. pt-online-schema-change (pt-osc)

61. pt-online-schema-change (pt-osc)
• A tool, performing non-blocking upgrades
• With TOI
30

62. pt-online-schema-change (pt-osc)
• A tool, performing non-blocking upgrades
• Creates a copy of table with altered definition
30

63. pt-online-schema-change (pt-osc)
• A tool, performing non-blocking upgrades
• Creates a copy of table with altered definition
• Creates triggers which will copy modified rows
30

64. pt-online-schema-change (pt-osc)
• A tool, performing non-blocking upgrades
• Creates a copy of table with altered definition
• Creates triggers which will copy modified rows
• Starts copying data in chunks
• Absolutely under control
• Can be paused or stopped
–max-flow-ctl
30

65. pt-online-schema-change (pt-osc)
• A tool, performing non-blocking upgrades
• Creates a copy of table with altered definition
• Creates triggers which will copy modified rows
• Starts copying data in chunks
• All rows already in the table are copied in chunks
• Newly modified rows are copied using triggers
30

66. pt-online-schema-change (pt-osc)
• A tool, performing non-blocking upgrades
• Creates a copy of table with altered definition
• Creates triggers which will copy modified rows
• Starts copying data in chunks
• Once copy is complete, drops the table
30

67. pt-online-schema-change (pt-osc)
• A tool, performing non-blocking upgrades
• Creates a copy of table with altered definition
• Creates triggers which will copy modified rows
• Starts copying data in chunks
• Once copy is complete, drops the table
• Renames the copy into the original table name
30

68. pt-osc Advantages
• DDL is safe and non-blocking
31

69. pt-osc Disadvantages
• Works only with InnoDB tables
• Increases IO load even for inplace operations
• Conflicts with already existing triggers
• Unless you use PXC >= 5.7
• Foreign keys updates are not effectively safe
32

70. How to Use pt-osc
• Study pt-osc options
• --max-flow-ctl
• Set appropriate limits
• Make sure wsrep OSU method is TOI
• Run pt-osc
33

71. Which Method to Use?
Will DDL be fast?
• CREATE DATABASE
• CREATE TABLE
• DROP INDEX
• Any ALTER on small tables
• Other
34

72. Which Method to Use?
Will DDL be fast?
Yes Use TOI
34

73. Which Method to Use?
Will DDL be fast?
Yes Use TOI
No Evaluate if you can use pt-osc
• Operation on the InnoDB table
• Table has no triggers or PXC >= 5.7
• Table is not referenced by a foreign key
• You can tolerate increased IO
34

74. Which Method to Use?
Will DDL be fast?
Yes Use TOI
No Evaluate if you can use pt-osc
Yes Use pt-osc
34

75. Which Method to Use?
Will DDL be fast?
Yes Use TOI
No Evaluate if you can use pt-osc
Yes Use pt-osc
No Use RSU
• Stop all write traffic on the node
• Stop all write traffic to the modified table
• Make sure to upgrade on all nodes
34

76. Conclusion
• Use TOI whenever possible
• Then use pt-osc
• RSU is a last resort
35

77. More information
Galera Cluster
Percona XtraDB Cluster
MariaDB Galera Cluster
pt-online-schema-change
36

78. Thank you!
www.slideshare.net/SvetaSmirnova
twitter.com/svetsmirnova
github.com/svetasmirnova
37