The document discusses synchronizing data between MySQL and MongoDB using Tungsten Replicator. Tungsten Replicator allows data to be replicated from a MySQL database to MongoDB in near real-time. The document provides examples of data being inserted and updated in MySQL, and then appearing in MongoDB through the replication process. It also discusses security features and how to install Tungsten Replicator for basic master-slave replication between MySQL and MongoDB databases.

1. Synchronize your data
between MySQL
and MongoDB
using Tungsten Replicator
Giuseppe Maxia, Director of QA
Continuent, Inc
©Continuent 2013
Tuesday, October 15, 13
1

2. About me
•
Giuseppe Maxia, a.k.a. "The Data Charmer"
• Director of Quality Assurance, Continuent, Inc
• 25+ years development and DB experience
• Long timer MySQL community member.
• Oracle ACE Director
• Blog: http://datacharmer.blogspot.com
• Twitter: @datacharmer
©Continuent 2013
Tuesday, October 15, 13
2
2

3. Introducing Continuent
•
The leading provider of clustering and
replication for open source DBMS
•
Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Flexible, high-performance data
movement
©Continuent 2013
Tuesday, October 15, 13
3
3

4. A Review of Tungsten Replicator
©Continuent 2013
Tuesday, October 15, 13
4
4

5. Tungsten Replicator Overview
Master
Replicator
Download
transactions
via network
DBMS
Logs
(Transactions + Metadata)
Slave
Replicator
Apply using JDBC
©Continuent 2013
Tuesday, October 15, 13
THL
THL
(Transactions + Metadata)
5
5

6. Master Replication Service
Pipeline
Stage
Extract Filter
Stage
Apply
Extract Filter
Apply
tcp/ip
Binlog
MySQL
Master
©Continuent 2013
Tuesday, October 15, 13
Slave
Replicators
Transaction
History Log
In-Memory
Queue
6
6

7. Slave Replication Service
Pipeline
Stage
Apply
Extract Filter
Stage
Apply
Extract Filter
Apply
tcp/ip
Extract Filter
Stage
Master
Replicator
Transaction
History Log
©Continuent 2013
Tuesday, October 15, 13
In-Memory
Queue
Slave
DBMS
7
7

8. master-slave
MySQL
Oracle
fan-in slave
Oracle
MySQL
all-masters
Heterogeneous
Oracle
MySQL
MySQL
Oracle
star
©Continuent 2013
Tuesday, October 15, 13
8

9. MongoDB in a nutshell
©Continuent 2013
Tuesday, October 15, 13
9
9

10. What is MongoDB
•
•
•
•
•
•
•
©Continuent 2013
Tuesday, October 15, 13
A non-relational database
A document-oriented database
Schema-free
Open source
High performance
Scalable
Developer-friendly (sort of)
10
10

11. What is MongoDB good for?
•
•
•
Storing large amount of unrelated data
•
IT IS NOT a drop-in replacement for a
relational database
©Continuent 2013
Tuesday, October 15, 13
Data that can't be constrained in a schema
Complement to relational data
11
11

12. Relational vs. document
person
p_id name age
1
Joe
30
2
Fred
23
3
Jack
26
p_id
d_id
4
Sue
25
1
2
5
Pete
32
2
2
3
1
4
3
5
1
department
d_id
name
1
sales
2
dev
3
support
©Continuent 2013
Tuesday, October 15, 13
pers_dept
Relational
12
12

13. Relational vs. document
person
p_id name age
1
Joe
30
2
Fred
23
3
Jack
26
p_id
d_id
4
Sue
25
1
2
5
Pete
32
2
2
3
1
4
3
5
1
department
d_id
name
1
sales
2
dev
3
support
©Continuent 2013
Tuesday, October 15, 13
pers_dept
Relational
13
13

14. Relational vs. document
person
p_id name age
1
Joe
30
2
Fred
23
3
Jack
26
p_id
d_id
4
Sue
25
1
2
5
Pete
32
2
2
3
1
4
3
5
1
department
d_id
name
1
sales
2
dev
3
support
©Continuent 2013
Tuesday, October 15, 13
pers_dept
Relational
14
14

15. Relational vs. document
person
p_id name age
1
Joe
30
2
Fred
23
3
Jack
26
p_id
d_id
4
Sue
25
1
2
5
Pete
32
2
2
3
1
4
3
5
1
department
d_id
name
1
sales
2
dev
3
support
©Continuent 2013
Tuesday, October 15, 13
pers_dept
Relational
15
15

16. Relational vs. document
person
p_id name age
1
Joe
30
2
Fred
23
3
Jack
26
p_id
d_id
4
Sue
25
1
2
5
Pete
32
2
2
3
1
4
3
5
1
department
d_id
name
1
sales
2
dev
3
support
©Continuent 2013
Tuesday, October 15, 13
pers_dept
Relational
16
16

17. Relational vs. document
person
p_id name age
1
Joe
30
2
Fred
23
3
Jack
26
p_id
d_id
4
Sue
25
1
2
5
Pete
32
2
2
3
1
4
3
5
1
department
d_id
name
1
sales
2
dev
3
support
©Continuent 2013
Tuesday, October 15, 13
pers_dept
Relational
17
17

18. Relational vs. document
person
_id
name
age
department
1
Joe
30
dev
2
Fred
23
dev
3
Jack
26
sales
4
Sue
25
support
5
Pete
32
sales
Document
©Continuent 2013
Tuesday, October 15, 13
18
18

19. How MongoDB keeps data
•
©Continuent 2013
Tuesday, October 15, 13
three levels:
•
•
•
dbs
collections
documents
19
19

20. MongoDB insertion demo
> show collections
>
> db.person.insert( {_id: 1, name: "Joe", age: 30, department:
"dev"})
> show collections
person
system.indexes
©Continuent 2013
Tuesday, October 15, 13
20
20

21. MongoDB insertion demo
> db.person.insert(
"dev"})
> db.person.insert(
"sales"})
> db.person.insert(
"support"})
> db.person.insert(
"sales"})
> db.person.find()
{ "_id" : 1, "name"
{ "_id" : 2, "name"
{ "_id" : 3, "name"
{ "_id" : 4, "name"
{ "_id" : 5, "name"
©Continuent 2013
Tuesday, October 15, 13
{_id: 2, name: "Fred", age: 23, department:
{_id: 3, name: "Jack", age: 26, department:
{_id: 4, name: "Sue", age: 25, department:
{_id: 5, name: "Pete", age: 30, department:
:
:
:
:
:
"Joe", "age" : 30, "department" : "dev" }
"Fred", "age" : 23, "department" : "dev" }
"Jack", "age" : 26, "department" : "sales" }
"Sue", "age" : 25, "department" : "support" }
"Pete", "age" : 30, "department" : "sales" }
21
21

22. MySQL to MongoDB basics
©Continuent 2013
Tuesday, October 15, 13
22
22

23. Replication from MySQL to MongoDB
•
•
•
•
•
•
•
©Continuent 2013
Tuesday, October 15, 13
Requires ROW-based-replication
Replication happens by table
There is no consolidation into "documents"
DDL commands are ignored
Statement commands are ignored
Column names become document attributes
enum and set columns are converted to
strings
23
23

24. First example of replication
# MySQL
create schema oneschema;
use oneschema ;
create table myfirst( num int not null primary key,
dt datetime,
ts timestamp,
going enum('yes', 'no'));
# MongoDB
> show dbs
local 0.078125GB
test 0.203125GB
tungsten_mysql2mongodb
0.203125GB
# NOTICE: no "oneschema"
©Continuent 2013
Tuesday, October 15, 13
24
24

25. Inserting data
# MySQL
insert into myfirst values (1, '2003-04-26 09:15:00', null, 'yes');
Query OK, 1 row affected (0.01 sec)
select * from myfirst;
+-----+---------------------+---------------------+-------+
| num | dt
| ts
| going |
+-----+---------------------+---------------------+-------+
|
1 | 2003-04-26 09:15:00 | 2013-10-14 19:39:38 | yes
|
+-----+---------------------+---------------------+-------+
1 row in set (0.00 sec)
©Continuent 2013
Tuesday, October 15, 13
25
25

26. Checking results in MongoDB
# MongoDB
> show dbs
local 0.078125GB
oneschema 0.203125GB
test 0.203125GB
tungsten_mysql2mongodb
0.203125GB
> use oneschema
switched to db oneschema
> show collections
myfirst
system.indexes
> db.myfirst.find()
{ "_id" : ObjectId("525c2c5af5d9ca820fcee01d"), "num" : "1", "dt" :
"2003-04-26 11:15:00.0", "ts" : "2013-10-14 19:39:38.0", "going" :
"yes" }
©Continuent 2013
Tuesday, October 15, 13
26
26

27. Another interesting insertion
#MySQL
create table t1(_id int not null primary key, c char(10));
insert into t1 values (1, 'abc');
select * from t1;
+-----+------+
| _id | c
|
+-----+------+
|
1 | abc |
+-----+------+
1 row in set (0.00 sec)
# MongoDB
> db.t1.find()
{ "_id" : "1", "c" : "abc" }
©Continuent 2013
Tuesday, October 15, 13
27
27

28. More insertions
# MySQL
insert into t1 values (2,'def'), (3,'ghi'), (4,'jkl'), (5, 'mno');
Query OK, 4 rows affected (0.01 sec)
Records: 4 Duplicates: 0 Warnings: 0
select * from t1;
+-----+------+
| _id | c
|
+-----+------+
|
1 | abc |
|
2 | def |
|
3 | ghi |
|
4 | jkl |
|
5 | mno |
+-----+------+
5 rows in set (0.00 sec)
©Continuent 2013
Tuesday, October 15, 13
28
28

29. More insertions
# MongoDB
>
{
{
{
{
{
db.t1.find()
"_id" : "1",
"_id" : "2",
"_id" : "3",
"_id" : "4",
"_id" : "5",
©Continuent 2013
Tuesday, October 15, 13
"c"
"c"
"c"
"c"
"c"
:
:
:
:
:
"abc"
"def"
"ghi"
"jkl"
"mno"
}
}
}
}
}
29
29

30. Update and delete as seen on master
update t1 set c = 'ZZZ' where _id = 3;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
delete from t1 where _id=2;
Query OK, 1 row affected (0.00 sec)
select * from t1;
+-----+------+
| _id | c
|
+-----+------+
|
1 | abc |
|
3 | ZZZ |
|
4 | jkl |
|
5 | mno |
+-----+------+
4 rows in set (0.00 sec)
©Continuent 2013
Tuesday, October 15, 13
30
30

31. e!ects of update and delete on the
slave
# MongoDB
>
{
{
{
{
db.t1.find()
"_id" : "1",
"_id" : "3",
"_id" : "4",
"_id" : "5",
©Continuent 2013
Tuesday, October 15, 13
"c"
"c"
"c"
"c"
:
:
:
:
"abc"
"ZZZ"
"jkl"
"mno"
}
}
}
}
31
31

32. Overview of Tungsten installer
©Continuent 2013
Tuesday, October 15, 13
32
32

33. Overview of Installation Process
1. Set up hosts
2. Prepare MySQL replicas
3. Download software
4. Install using tpm
Amazon Setup:
https://docs.continuent.com/wiki/display/TEDOC/
Preparing+EC2+Servers
©Continuent 2013
Tuesday, October 15, 13
33
33

34. How tungsten-installer Works for
Basic Master/Slave Deployment
Staging copy
of files
db1
db2
check prereqs
copy code
configure
©Continuent 2013
Tuesday, October 15, 13
db3
34
34

35. Tungsten master/slave replication
alpha
THL
THL
host1
alpha
host2
alpha
THL
host3
installer
©Continuent 2013
Tuesday, October 15, 13
35
35

36. Bi-directional replication
alpha
alpha
bravo
bravo
host1
host2
installer
Install all master and slave services on all hosts at once
©Continuent 2013
Tuesday, October 15, 13
36
36

37. 4 nodes all-masters
alpha
alpha
bravo
host1
bravo
charlie
charlie
delta
host2
delta
alpha
bravo
©Continuent 2013
Tuesday, October 15, 13
bravo
charlie
charlie
delta
host3
alpha
delta
host4
37
37

38. Tungsten security layer
•
Tra!c encryption:
Tuesday, October 15, 13
all data in transit (transaction history logs, or THL) is
encrypted using SSL
•
•
•
©Continuent 2013
•
all administrative tra!c is encrypted with SSL
Transparent to the user
Independent of the database server (works also for
heterogeneous replication)
38
38

39. Tungsten replicator without security
THL
alpha
THL
alpha
alpha
host2
plain text
THL
host1
THL
host4
host3
master
slave
©Continuent 2013
Tuesday, October 15, 13
alpha
replicator
services
39
39

40. Tungsten Replicator with security
SSL
alpha
THL
alpha
THL
THL
alpha
host2
SSL
host1
SSL
SSL
slave
©Continuent 2013
Tuesday, October 15, 13
alpha
host4
host3
master
THL
replicator
services
40
40

41. Installing Master/Slave Replication ...
alpha
THL
THL
host1
alpha
host2
alpha
THL
host3
©Continuent 2013
Tuesday, October 15, 13
41
41

42. master/slave using tpm
./tools/tpm install alpha
--topology=master-slave
--home-directory=/opt/continuent/replicator
--replication-user=tungsten
--replication-password=secret
--master=host1
--slaves=host2,host3,host4
--start
©Continuent 2013
Tuesday, October 15, 13
42
42

43. Installing Master/Slave Replication
with MongoDB
alpha
alpha
host1
host2
alpha
alpha
host3
©Continuent 2013
Tuesday, October 15, 13
host4
43
43

44. master/slave with MongoDB
./tools/tpm configure mysql2mongodb
--enable-heterogenous-service=true
--topology=master-slave
--master=host1
--replication-user=tungsten
--replication-password=secret
--slaves=host2,host3,host4
--home-directory=$MYSQL_DEPLOY
--start-and-report
./tools/tpm configure mysql2mongodb
--hosts=host4
--datasource-type=mongodb
--replication-port=$MONGODB_PORT
./tools/tpm install
©Continuent 2013
Tuesday, October 15, 13
44
44

45. Installing Fan-In Replication
alpha
bravo
host1
host2
alpha
charlie
bravo
host3
©Continuent 2013
Tuesday, October 15, 13
host4
charlie
45
45

46. fan-in using tpm
./tools/tpm install many_towns
--replication-user=tungsten
--replication-password=secret
--home-directory=/opt/continuent/replication
--masters=host1,host2,host3
--slaves=host4
--master-services=alpha,bravo,charlie
--topology=fan-in
--start
©Continuent 2013
Tuesday, October 15, 13
46
46

47. Installing Fan-In Replication with
MongoDB
alpha
bravo
host1
host2
alpha
bravo
host3
©Continuent 2013
Tuesday, October 15, 13
charlie
charlie
host4
47
47

48. fan-in with MongoDB
./tools/tpm configure mysql2mongodb
--enable-heterogenous-service=true
--topology=fan-in
--masters=host1,host2,host3
--master-services=alpha,bravo,charlie
--slaves=host4
--replication-user=tungsten
--replication-password=secret
--home-directory=$MYSQL_DEPLOY
--datasource-type=mysql
--start-and-report
./tools/tpm configure mysql2mongodb
--hosts=host4
--datasource-type=mongodb
--replication-port=$MONGODB_PORT
./tools/tpm install
©Continuent 2013
Tuesday, October 15, 13
48
48

49. Install Multi-Master replication
alpha
alpha
bravo
host1
bravo
charlie
charlie
host2
alpha
bravo
host3
©Continuent 2013
Tuesday, October 15, 13
charlie
49
49

50. multi-master using tpm
../tools/tpm install musketeers
--reset
--topology=all-masters
--home-directory=/opt/continuent/replicator
--replication-user=tungsten
--replication-password=secret
--masters=host1,host2,host3
--master-services=alpha,bravo,charlie
--start
©Continuent 2013
Tuesday, October 15, 13
50
50

51. Install Multi-Master replication with
Mongodb
alpha
alpha
bravo
host1
bravo
charlie
charlie
alpha
host3
©Continuent 2013
Tuesday, October 15, 13
alpha
bravo
bravo
charlie
charlie
host2
host4
51
51

52. multi-master with MongoDB
./tools/tpm configure mysql2mongodb
--enable-heterogenous-service=true
--topology=all-masters
--masters=host1,host2,host3
--slaves=host1,host2,host3,host4
--master-services=alpha,bravo,charlie
--replication-user=tungsten
--replication-password=secret
--home-directory=$MYSQL_DEPLOY
--datasource-type=mysql
--start-and-report
./tools/tpm configure mysql2mongodb
--hosts=host4
--datasource-type=mongodb
--replication-port=$MONGODB_PORT
./tools/tpm install
©Continuent 2013
Tuesday, October 15, 13
52
52

53. MongoDB or TokuMX
•
TokuMX is a drop-in replacement for
MongoDB
•
•
•
Open source project, developed by TokuTek
©Continuent 2013
Tuesday, October 15, 13
https://github.com/Tokutek/mongo
it includes
•
•
•
•
better indexing
row-level locking (MongoDB locks at db level)
transactions
better compression
53
53

54. DEMO:
MongoDB
and multi master
installation
©Continuent 2013
Tuesday, October 15, 13
54
54

55. Joining the Community
©Continuent 2013
Tuesday, October 15, 13
55
55

56. Tungsten Replicator is Open Source
•
Project home:
http://code.google.com/p/tungsten-replicator/
•
Log bugs, "nd builds, post in replicator discussion
group
•
Documentation:
https://docs.continuent.com/wiki/display/TEDOC/
Tungsten+Documentation+Home
https://docs.continuent.com/wiki/display/TEDOC/
Deploying+MongoDB+Replication
©Continuent 2013
Tuesday, October 15, 13
56
56

57. 560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com
Our Blogs:
http://scale-out-blog.blogspot.com
http://datacharmer.blogspot.com
http://www.continuent.com/news/blogs
Continuent Web Page:
http://www.continuent.com
Tungsten Replicator 2.1:
http://code.google.com/p/tungsten-replicator
©Continuent 2012.
Tuesday, October 15, 13
57