How Safe is Asynchronous Master-Master Setup?
•
0 likes•438 views
Presented at Percona MySQL Tech Day on September 10, 2020: https://www.percona.com/tech-days#mysql It is common knowledge that built-in asynchronous active-active replication is not safe. I remember times when the official MySQL User Reference Manual stated that such an installation is not recommended for production use. Some experts repeat this claim even now. While this statement is generally true, I worked with thousands of shops that successfully avoided asynchronous replication limitations in active-active setups. In this talk, I will show how they did it, demonstrate situations when asynchronous source-source replication is the best possible high availability option and beats such solutions as Galera or InnoDB Clusters. I will also cover common mistakes, leading to disasters.
Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Similar to How Safe is Asynchronous Master-Master Setup?
Similar to How Safe is Asynchronous Master-Master Setup? (20)
More from Sveta Smirnova
More from Sveta Smirnova (20)
Recently uploaded
Recently uploaded (20)
How Safe is Asynchronous Master-Master Setup?
- 1. How Safe is Asynchronous Active-Active Setup in MySQL September 10, 2020 Sveta Smirnova
- 2. • MySQL Support engineer • Author of • MySQL Troubleshooting • JSON UDF functions • FILTER clause for MySQL • Speaker • Percona Live, OOW, Fosdem, DevConf, HighLoad... Sveta Smirnova 2
- 3. •What Happens with 2 Active Nodes? •Why to Write to Multiple Nodes? •Galera, PXC, InnoDB Cluster •How to Setup Safe Active-Active Table of Contents 3
- 4. Source Replica ← Initiates Asynchronous Built-In Replication 4
- 5. Source Replica ← Initiates ← Requests a packet Asynchronous Built-In Replication 4
- 6. Source Sends the packet → Replica ← Initiates ← Requests a packet Asynchronous Built-In Replication 4
- 7. Source Sends the packet → Replica ← Initiates ← Requests a packet ... ? Asynchronous Built-In Replication 4
- 8. Node1 Initiates → Node2 ← Initiates Asynchronous Two Active Nodes 5
- 9. Node1 Initiates → Requests a packet → Node2 ← Initiates ← Requests a packet Asynchronous Two Active Nodes 5
- 10. Node1 Initiates → Requests a packet → Sends the packet → Node2 ← Initiates ← Requests a packet ← Sends the packet Asynchronous Two Active Nodes 5
- 11. Node1 Initiates → Requests a packet → Sends the packet → ... ? Node2 ← Initiates ← Requests a packet ← Sends the packet ... ? Asynchronous Two Active Nodes 5
- 12. What Happens with 2 Active Nodes?
- 13. Node1 INSERT SET id=42 Node2 INSERT SET id=42 INSERT 7
- 14. Node1 INSERT SET id=42 Sends the update → Node2 INSERT SET id=42 ← Sends the update INSERT 7
- 15. Node1 INSERT SET id=42 Sends the update → Receives update → Node2 INSERT SET id=42 ← Sends the update ← Receives the update INSERT 7
- 16. Node1 INSERT SET id=42 Sends the update → Receives update → Duplicate key error! Node2 INSERT SET id=42 ← Sends the update ← Receives the update Duplicate key error! INSERT 7
- 17. Node1 DELETE WHERE id=42 Node2 DELETE WHERE id=42 DELETE 8
- 18. Node1 DELETE WHERE id=42 Sends the update → Node2 DELETE WHERE id=42 ← Sends the update DELETE 8
- 19. Node1 DELETE WHERE id=42 Sends the update → Receives update → Node2 DELETE WHERE id=42 ← Sends the update ← Receives the update DELETE 8
- 20. Node1 DELETE WHERE id=42 Sends the update → Receives update → Key not found error! Node2 DELETE WHERE id=42 ← Sends the update ← Receives the update Key not found error! DELETE 8
- 21. Node1 UPDATE SET i=42 Node2 UPDATE SET i=25 UPDATE 9
- 22. Node1 UPDATE SET i=42 Sends the update → Node2 UPDATE SET i=25 ← Sends the update UPDATE 9
- 23. Node1 UPDATE SET i=42 Sends the update → Receives update → Node2 UPDATE SET i=25 ← Sends the update ← Receives the update UPDATE 9
- 24. Node1 UPDATE SET i=42 Sends the update → Receives update → i=25 Node2 UPDATE SET i=25 ← Sends the update ← Receives the update i=42 UPDATE 9
- 25. Why to Write to Multiple Nodes?
- 26. • Each node for its own purpose Application 1 Application 2 Data Distribution 11
- 27. • Multiple machines for better scalability Node 1 Node 2 [Read] Scale 12
- 28. • Nodes in different geographical regions Belgium Brazil GEO Distribution 13
- 29. • Waits when the active node dies • Then accepts writes Active Hot Standby Hot Standby 14
- 30. Galera, PXC, InnoDB Cluster
- 31. Node1 UPDATE ... Node2,3,... Fully Synchronous Cluster 16
- 32. Node1 UPDATE ... Sends the packet → Node2,3,... Fully Synchronous Cluster 16
- 33. Node1 UPDATE ... Sends the packet → Waits Node2,3,... ← Receives the packet Fully Synchronous Cluster 16
- 34. Node1 UPDATE ... Sends the packet → Waits Waits Node2,3,... ← Receives the packet Applies changes Fully Synchronous Cluster 16
- 35. Node1 UPDATE ... Sends the packet → Waits Waits Waits Node2,3,... ← Receives the packet Applies changes ← Confirms Fully Synchronous Cluster 16
- 36. Node1 UPDATE ... Sends the packet → Waits Waits Waits Receives answer → Node2,3,... ← Receives the packet Applies changes ← Confirms Fully Synchronous Cluster 16
- 37. Node1 UPDATE ... Sends the packet → Waits Waits Waits Receives answer → Ready for new update Node2,3,... ← Receives the packet Applies changes ← Confirms Fully Synchronous Cluster 16
- 38. • Speed of the slowest member Synchronization Penalty 17
- 39. • Speed of the slowest member • Writes or reads should wait Synchronization Penalty 17
- 40. • Speed of the slowest member • Writes or reads should wait • Tunable Synchronization Penalty 17
- 41. • Speed of the slowest member • Writes or reads should wait • Tunable • Galera • wsrep sync wait • InnoDB Cluster • group replication consistency Synchronization Penalty 17
- 42. • Synchronization takes too much time • Nodes often disconnect GEO Replication is Painful 18
- 43. How to Setup Safe Active-Active
- 44. • Business logic • Servers • Databases • Tables • Rows Divide and Rule 20
- 45. • Write only to one server at a time Node 1 (Active) Node 2 (Hot Standby) Servers 21
- 46. • Write only to one server at a time • Switch when needed Node 1 (Dead) Node 2 (Active) Servers 21
- 47. • Write only to one server at a time • Switch when needed Node 1 (Hot Standby) Node 2 (Active) Servers 21
- 48. • Each server writes to its own set of tables Node 1 Node 2 D1 D2 D3 D4 D1 D2 D3 D4 Databases and Tables 22
- 49. • Agreement on which rows Node 1 Node 2 Row 1 Row 2 Row 3 Row 4 Row 1 Row 2 Row 3 Row 4 Rows 23
- 50. • Agreement on which rows • E.g. AUTO INCREMENT • auto increment increment=NUMBER OF SERVERS • auto increment offset = server id • Custom index • Create your own unique pattern for each node Rows 23
- 51. • Hot Standby • Easy to setup and maintain • Separating writes by database/table • Comparatively easy to setup and maintain • Separating writes by row • Should be used very carefully Conclusion 24
- 52. MySQL User Reference Manual Why MySQL Replication Fails, and How... MySQL High Availability More information 25