Geographically Distributed Multi-Master MySQL Clusters
•
0 likes•1,449 views
In this webinar, we discuss the multi-master capabilities of Continuent Tungsten to help you build and manage systems that spread data across multiple sites. We cover important topics such as setting up large scale topologies, handling failures, and how to handle data privacy issues like removing personally identifiable information or handling privacy law restrictions on data movement. We will conclude with a live demonstration of a distributed MySQL solution with Continuent Tungsten clusters working across multiple Amazon Web Services (AWS) availability zones and regions.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Geographically Distributed Multi-Master MySQL Clusters
Similar to Geographically Distributed Multi-Master MySQL Clusters (20)
More from Continuent
More from Continuent (20)
Recently uploaded
Recently uploaded (20)
Geographically Distributed Multi-Master MySQL Clusters
- 1. ©Continuent 2014 Geographically Distributed Multi-Master MySQL Clusters Robert Hodges, CEO Jeff Mace, Director of Services
- 2. ©Continuent 2014 Introducing Continuent 2 • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement
- 3. ©Continuent 2014 Quick Continuent Facts • Largest Tungsten clustering installation processes up to 700M transactions daily • Over 100 customers including many household names • Multiple multi-site, multi-master deployments in production • ... And more are on the way 3
- 4. ©Continuent 2014 4 Tungsten Clustering in Two Minutes
- 5. ©Continuent 2014 Tungsten Database-as-a-Service 5 Tungsten clusters combine off- the-shelf open source DBMS servers into data services with: ! • 24x7 data access • Scaling of load on replicas • Simple management commands ! ...without app changes or data migration Amazon US West apache /php GonzoPortal.com Connector Connector
- 6. ©Continuent 2014 Cluster Detailed View 6 Application Tungsten Connector Application Tungsten Connector Replicator Db2 Replicator Db3Db1 Replicator Slave SlaveMaster Manager Manager Manager MonitoringandControl MonitoringandControl Data Service: sjc
- 7. ©Continuent 2014 7 Tungsten Replication in Four Minutes
- 8. ©Continuent 2014 Tungsten Replicator Overview 8 Master (Transactions + Metadata) Slave THL DBMS Logs Replicator (Transactions + Metadata) THLReplicator Download transactions via network Apply using JDBC
- 9. ©Continuent 2014 Replication Pipelines 9 Extract Filter Apply Stage Extract Filter Apply Stage Pipeline Master DBMS Transaction History Log In-Memory Queue Slave DBMS Extract Filter Apply Stage
- 10. ©Continuent 2014 Multiple Services Per Replicator 10 NYC Replicator London Service nyc Frankfurt Replicator Service fra Replicator Service nyc Service fra
- 11. ©Continuent 2014 Using Services for Multi-Master Replication 11 New York London Replicator lhr (master) jfk (slave) Replicator lhr (slave) jfk (master)
- 12. ©Continuent 2014 12 Introducing Multi-Site Multi-Master Clustering
- 13. ©Continuent 2014 Why Multi-Site? Why Multi-Master? 13 • Get data close to global applications • Avoid latency cost of commit between sites • Keep running during intermittent network failures between sites • Keep running if a site fails • Enable simple recovery when it comes back
- 14. ©Continuent 2014 Multi-Site Multi-Master Clustering: the Marketecture 14 No app or data coupling across regions ApplicationApplications ApplicationApplications Amazon Region Amazon Region Live data and apps in all regions Multi-AZ deployment Async replication between sites Run in Amazon or on-premises
- 15. ©Continuent 2014 Detailed Topology 15 US-West-1 US-East-1 master slave east1 east1 west west1 master slave Important Note: Continuent recommends 3 nodes for clusters
- 16. ©Continuent 2014 16 Demo Setting up Multi-Master Clustering across Sites US-East-1 US-West-1
- 17. ©Continuent 2014 Puppet Installation and Apply 17 $> puppet module install continuent/tungsten $> puppet apply ~/tungsten.pp See https://github.com/continuent/ continuent-vagrant for more examples
- 18. ©Continuent 2014 18 Failures in Multi-Master Topologies
- 19. ©Continuent 2014 Best Practice Deployment in Amazon 19 Multi-AZ clustering with asynchronous multi- master across regions US-East-1 AZAZ AZ AZ EU- West-1 AZ AZ AZ US- West-1 AZ AZ AZ AZAZ AZ AZAZ Availability ZoneRegion AZ AZ AZ
- 20. ©Continuent 2014 Normal Multi-Master Configuration 20 master slave master slaveeast1 east1 west west1 US-West-1 US-East-1
- 21. ©Continuent 2014 Failed master 21 master master master slave master X east1 east1 west west1 US-West-1 US-East-1
- 22. ©Continuent 2014 Failed Slave 22 master slave master slave X east1 east1 west west1 US-West-1 US-East-1
- 23. ©Continuent 2014 Failed Multi-Master Replicator 23 master slave master slaveeast1 east1 west west1 X US-West-1 US-East-1
- 24. ©Continuent 2014 Full-Site Failure 24 master slave master slaveeast1 east1 west west1 X US-West-1 US-East-1
- 25. ©Continuent 2014 25 Enabling Data Replication Policies with Filters
- 26. ©Continuent 2014 What Are Replication Filters? 26 Extract Filter Apply Stage Extract Filter Apply Stage Master Pipeline Master DBMS Transaction History Log In-Memory Queue Inspect Change Drop
- 27. ©Continuent 2014 Problem: Handling PII Data 27 CREATE TABLE `personal_info` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userid` varchar(250) DEFAULT NULL, `personal_code` varchar(250) DEFAULT NULL, `birth_date` date DEFAULT NULL, `email` varchar(250) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 Requirement 1: Do not replicate personally identifiable information
- 28. ©Continuent 2014 Problem: Application-Specific Rules 28 mysql> select * from personal_info; +----+--------+---------------+------------+-----------------+ | id | userid | personal_code | birth_date | email | +----+--------+---------------+------------+-----------------+ | 1 | fre11 | secret | 1985-01-16 | bobr@gmail.com | | 2 | bill | verysecret | 1995-01-16 | billk@gmail.com | +----+--------+---------------+------------+-----------------+ Requirement 2: prevent userid ‘bill’ from replicating
- 29. ©Continuent 2014 Adding Filters to Meet Requirements 29 Extract Filter(s) Apply Stage Extract Filter Apply Stage Master Pipeline Master DBMS Transaction History Log In-Memory Queue colnames killbill dropcolumns dropcolumns.json
- 30. ©Continuent 2014 Creating a dropcolumn.json File [ { "schema": "test", "table": "personal_info", "columns": [ "personal_code", "birth_date", "email" ] } ] 30 Table name Columns to delete
- 31. ©Continuent 2014 Writing a Custom killbill.js Filter function filter(event) {! data = event.getData();! if(data != null) {! for (i = 0; i < data.size(); i++) {! d = data.get(i);! if(d != null && d instanceof com.continuent.tungsten.replicator.dbms! .RowChangeData) {! rowChanges = d.getRowChanges();! for(j = 0; j < rowChanges.size(); j++) {! oneRowChange = rowChanges.get(j);! schema = oneRowChange.getSchemaName();! table = oneRowChange.getTableName();! if (schema == "test" && table == "personal_info") {! columnValues = oneRowChange.getColumnValues();! for (row = 0; row < columnValues.size(); row++) {! raw = columnValues.get(row).get(1).getValue();! user = new java.lang.String(raw);! if (user.equals("bill")) {! logger.info("Dropping transaction from bill!");! return null;! } } } } } } } }! 31 Drop transaction
- 32. ©Continuent 2014 Killbill.tpl file for Filter Installation # Drop transactions for bill. replicator.filter.killbill=com.continuent.tungsten.replicator. filter.JavaScriptF ilter replicator.filter.killbill.script=/opt/continuent/share/ killbill.js 32
- 33. ©Continuent 2014 Installing the Filters... cp dropcolumn.json /opt/continuent/share cp killbill.js /opt/continuent/share cp killbill.tpl /opt/tungsten/tungsten-replicator/samples/ conf/filters/default ! /opt/continuent/tungsten/tools/tpm update --svc-extractor-filters=colnames,killbill,dropcolumn --property=replicator.filter.dropcolumn.definitionsFile=/ opt/continuent/share/dropcolumn.json --start-and-report=true 33
- 34. ©Continuent 2014 Resulting Data on Slave(s) 34 mysql> select * from personal_info; +----+--------+---------------+------------+-------+ | id | userid | personal_code | birth_date | email | +----+--------+---------------+------------+-------+ | 1 | fre11 | NULL | NULL | NULL | +----+--------+---------------+------------+-------+ 1 row in set (0.00 sec) ! Requirement 2: No bill! Requirement 1: PII columns cleared
- 36. ©Continuent 2014 How Do I Get Started? 36 • Documentation for all products http://docs.continuent.com • Download registration for Tungsten clustering: http://www.continuent.com/downloads/software • Downloads for open source Tungsten Replicator http://code.google.com/p/tungsten-replicator/ • Vagrant project for setting up test topologies https://github.com/continuent/continuent-vagrant/tree/master/ examples/MSMM
- 37. ©Continuent 2014 Continuent Web Page: http://www.continuent.com ! Tungsten Replicator: http://code.google.com/p/tungsten-replicator Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.org/blog http://www.continuent.com/news/blogs 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com