Global data access can greatly expand the reach of your business. Continuent's multi-site multi-master (MSMM) solutions enable applications to accept write traffic in multiple locations across on-premises and vCloud Air. As an example, this includes the following real-world, business-critical use cases: - Improve performance for globally distributed users registering hardware devices by permitting updates on the geographically closest site - Ensure availability of credit card processing by spreading transaction processing across two or more sites. Users can still process credit card transactions if a single site is unavailable to them for any reason, including end-user Internet routing problems - Enable business continuity by using multi-master updates on different hosting providers for service scalability, personalization and software upgrades of GPS devices. Individual Continuent clusters already provide excellent single-site database availability and performance. In this webinar we review the benefits of combining multiple Continuent clusters into a global multi-site multi-master (MSMM) topology for: - Optimizing your installation for MSMM - Optimizing your application for MSMM - Monitoring and administration - Failover and recovery of individual servers or entire locations.

1. © 2014 VMware Inc. All rights reserved.
Geographically Distributed
Multi-Master MySQL Clusters
Featuring VMware Continuent
Jeff Mace
2/10/2015

2. VMware Continuent Quick Introduction
History Products
VMware Continuent
Industry-leading clustering and
replication for open source DBMS
Clustering – Commercial-grade
HA, performance scaling and
data management for MySQL
Replication – Flexible, high-
performance data movement
2004 Continuent established in
USA
2009 3rd Generation Continuent
Tungsten (aka VMware
Continuent) ships
2014 100+ customers running
business critical applications
Oct 2014 Acquisition by VMware: Now
part of the vCloud Air
Business Unit
Oct 2015 Continuent solutions available
through VMware sales

3. VMware Continuent Facts
Business Critical Deployment Examples
High Availability for
MySQL
Largest cluster deployment performs 800M+ transactions/
day on 275 TB of relational data
Business Continuity Cross-site cluster topologies widely deployed including
Primary/DR and Multi-Master
High Performance
Replication
Largest installations transfer billions of transactions daily
using high speed, parallel replication
Heterogeneous
Integration
Customers replicate from MySQL to Oracle, Hadoop,
Redshift, Vertica and others
Real-time Analytics Optimized data loading for data warehouses with
deployments of up to 200 MySQL masters feeding
Hadoop

4. Select VMware Continuent Customers

5. The Case For Many Sites

6. Registering and Updating Distributed Devices
• There are many handheld and portable devices that move between
cities and regions.
• These devices need constant access to pull data or updates.
• DNS routes the user to an available site with the lowest latency
• Each site has full access to modify data for the user
• Benefit: Transactions are always processed at the fastest location

7. Europe Asia/Pacific
New Jersey1ms
2ms
1.5ms

8. Credit Card Transaction Processing
• Application connectivity to a particular location isn’t guaranteed
• Processing apps use a list of potential sites
• Connectivity to each site is attempted until successful
• The application proceeds as normal with the successful site
• Benefit: Automatic failover to any data center by the application

9. Europe Asia/Pacific
New Jersey3
2
1

10. Remote Office Connectivity
• Local sites potentially subject to prolonged outages
• Business processing continues at other sites
• Replication resumes when connectivity is restored
• Benefit: Each data center is able to run independently

11. Europe Asia/Pacific
New Jersey

12. Introducing Multi-Site
Multi-Master (MSMM)

13. CROSS-REGION
REPLICATION
Public Internet or VMware NSX Secure Gateway
vCLOUD AIR
VIRTUAL DATA CENTER
ON-PREMISES
DATA CENTER
DB2.CA
SLAVE
DB1.CA
MASTER
DB3.CA
SLAVE
Continuent Connector Continuent Connector
DB2.NJ
SLAVE
DB1.NJ
MASTER
DB3.NJ
SLAVE
Continuent Connector Continuent Connector
Asynchronous
Multi-Master
MSMM Topology

14. MSMM Components
• Each location runs an independent master/slave cluster.
• The cluster is responsible for ensuring local availability.
• An additional replicator is installed on every server.
• The additional replicator replicates data from one or more remote
locations to the local server.
• Replication data is never written to the slave binary log.

15. MSMM Replication
• Each node runs an additional replicator to apply data
from remote data centers.
• Each cluster should run with 3 nodes (1 is hidden).
vCLOUD AIR
VIRTUAL DATA CENTER
DB2.NJ
DB1.NJ
CA
NJ
CA
NJ
NJ
MASTER
ON-PREMISES
DATA CENTER
DB2.CA
DB1.CA
CA
NJ
CA
NJ
CA
MASTER

16. Local Failover
ON-PREMISES
DATA CENTER
DB2.CA
SLAVE
DB1.CA
MASTER
DB3.CA
SLAVE
Continuent Connector Continuent Connector
X
• The master has failed.
• The manager on a remaining server identifies the failure.

17. Local Failover
ON-PREMISES
DATA CENTER
DB2.CA
MASTER
DB1.CA
SHUNNED
DB3.CA
SLAVE
Continuent Connector Continuent Connector
X
• The failed server is shunned.
• One of the slaves is promoted to master.
• The connectors send writes to the new master.

18. Local Failover
ON-PREMISES
DATA CENTER
DB2.CA
MASTER
DB1.CA
SLAVE
DB3.CA
SLAVE
Continuent Connector Continuent Connector
• An administrator checks the failed server and corrects the issue.
• They run a recover command to reintroduce the server.

19. MSMM Replication After Failover
• Each node runs an additional replicator to apply data
from remote data centers.
• Each cluster should run with 3 nodes (1 is hidden).
vCLOUD AIR
VIRTUAL DATA CENTER
DB2.NJ
DB1.NJ
CA
NJ
CA
NJ
NJ
MASTER
ON-PREMISES
DATA CENTER
DB2.CA
DB1.CA
CA
NJ
CA
NJ
CA
MASTER
X

20. Demo

21. The Case Against Many Sites
• It is difficult.
• The application, database and operations teams must all build with
multi-site multi-master in mind.
• The topology implies that no site will ever be fully consistent with all
locations. There will always be pending transactions.

22. Successful MSMM

23. Application Design
• Shard users/locations into different schemas if possible
• Design the application with heavy INSERT workloads
• Avoid UPDATE/DELETE statements of the same data from different
locations
• Move batch operations to a single location
• Limit the size of commits for batch operations

24. Database Design
• Use InnoDB or another ACID compliant table engine
• Use truly unique primary keys like auto_increment or UUID
• Enable auto_increment_increment and
auto_increment_offset to prevent conflicts between sites
• Limit the use of triggers or prepare them for row-based replication with
Tungsten Replicator.
• Use NTP and consistent time zones on every server

25. Operations Plan
• Handle DNS management with a global DNS system
• Configure 24/7 monitoring of all sites
• Schedule regular runs of pt-table-checksum
– Expect some level of differences due to the nature of MSMM
– Monitor results for an increasing number of differences
• Schedule regular backups in each location

26. Wrapping Up

27. Synchronous Clustering Fails Over Distance
• Synchronous replication adds server performance drag (Daniel Abadi,
Yale University)
• Global locks create exploding deadlock problems (Jim Gray, Microsoft)
• Strong consistency between DBMS requires some/all regions to stop
when network fails (CAP proof by Nancy Lynch, MIT)

28. Multi-Master is Best for Multi-Site Operation
Europe Asia/Pacific
New Jersey
Optimized
performance
for users
SQL transaction
processing
in any region
Local
high-availability
in any region
Continuous
updates across
regions

29. For more information:
Eero Teerikorpi
Sr. Director, Strategic Alliance
eteerikorpi@vmware.com
+1 (408) 431 3305
Robert Noyes
Alliance Manager, USA & Canada
rnoyes@vmware.com
+1 (650) 575-0958
Philippe Bernard
Alliance Manager, EMEA & APAC
pbernard@vmware.com
+41 79 347 1385