MySQL Document Store - How to replace a NoSQL database by MySQL without effort but with a lot gains ?

ROOUG, December 2017
MySQL Document Store
How to replace a NoSQL database by MySQL without effort but with a lot of
gains ?

Frédéric Descamps - MySQL Community Manager - Oracle
3 / 110

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purpose only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied up in
making purchasing decisions. The development, release and timing of any features or
functionality described for Oracle´s product remains at the sole discretion of Oracle.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
4 / 110

about.me/lefred
Who am I ?
5 / 110

Frédéric Descamps
@lefred
MySQL Evangelist
Hacking MySQL since 3.23
devops believer
living in Belgium 🇧🇪
http://lefred.be
6 / 110

RDBMS & MySQL
Relational Databases
7 / 110

Data Integrity
8 / 110

Data Integrity
normalization
9 / 110

Data Integrity
normalization
constraints (foreign keys, ...)
10 / 110

Data Integrity
normalization
Atomicity, Consistency, Isolation, Durability
11 / 110

Data Integrity
normalization
ACID compliant
12 / 110

Data Integrity
normalization
ACID compliant
transactions
13 / 110

Data Integrity
normalization
ACID compliant
transactions
SQL
14 / 110

Data Integrity
normalization
ACID compliant
transactions
SQL
powerfull query language
15 / 110

NoSQL & MySQL
NoSQL Databases
16 / 110

NoSQL or Document Store
Schemaless
17 / 110

Schemaless
no schema design, no normalization, no foreign keys, no data types, ...
18 / 110

Schemaless
very quick initial development
19 / 110

Schemaless
Flexible data structure
20 / 110

Schemaless
embedded arrays or objects
21 / 110

Schemaless
valid solution when natural data can't be modelized optimaly into a relational
model
22 / 110

Schemaless
model
objects persistance without the use of any ORM - mapping oobject-oriented
23 / 110

Schemaless
model
JSON
24 / 110

Schemaless
model
JSON
close to frontend
25 / 110

Schemaless
model
JSON
close to frontend
native in JS
26 / 110

Schemaless
model
JSON
close to frontend
native in JS
easy to learn
27 / 110

How DBAs see data
28 / 110

How DBAs see data How Developers see data
{
"GNP" : 249704,
"Name" : "Belgium",
"government" : {
"GovernmentForm" :
"Constitutional Monarchy, Federation",
"HeadOfState" : "Philippe I"
},
"_id" : "BEL",
"IndepYear" : 1830,
"demographics" : {
"Population" : 10239000,
"LifeExpectancy" : 77.8000030517578
},
"geography" : {
"Region" : "Western Europe",
"SurfaceArea" : 30518,
"Continent" : "Europe"
}
}
29 / 110

Relational Model
Higher upfront effort during design
stage
data modeling and normalization
physical schema definition
Database ensures integrity (invalid
data rejected)
Data types
Constraints (foreign keys etc.)
Relational Databases vs Document Store
Systems
30 / 110

Relational Model
Higher upfront effort during design
stage
data modeling and normalization
physical schema definition
Database ensures integrity (invalid
data rejected)
Data types
Constraints (foreign keys etc.)
Document Model
Lower upfront effort
data structure defined in
application code
Application responsible for data
integrity
possibility for bad data to creep
into the system (because of bugs,
application changes etc.)
Systems
31 / 110

Relational Model
Steeper learning curve
requires knowledge of SQL
Query language allows optimizer to
execute complex queries efficiently
(specially for JOINs)
Schema changes can be problematic
(ex upgrades)
Systems (2)
32 / 110

Relational Model
Steeper learning curve
requires knowledge of SQL
Query language allows optimizer to
execute complex queries efficiently
(specially for JOINs)
Schema changes can be problematic
(ex upgrades)
Document Model
Easier to get started with
Limited or no JOINs, simple optimizer -
complexity off-loaded to application
Simpler deployment, different versions
of the document can coexist
Systems (2)
33 / 110

DMBS or NoSQL ?
34 / 110

DMBS or NoSQL ?
Why not both ?
35 / 110

Developers:
[x] schemaless
[x] rapid prototying/simpler APIs
[x] document model
[x] transactions
Operations:
[x] performance
management/visibility
[x] robust replication, backup,
restore
[x] comprehensive tooling
ecosystem
[x] simpler application schema
upgrades
Business Owner:
[x] don't lose my data == ACID trx
[x] capture all my data =
extensible/schemaless
[x] product on schedule/time to
market = rapid developement
A solution for all
36 / 110

MySQL as Document Store
Requirements
37 / 110

MySQL as Document Store: requirements
38 / 110

1. support of JSON datatype
39 / 110

2. CRUD Operations
40 / 110

2. CRUD Operations
3. X-Plugin & X Protocol
41 / 110

2. CRUD Operations
extends and replaces the traditional MySQL protocol
42 / 110

2. CRUD Operations
supports asynchronous communication to the server
43 / 110

2. CRUD Operations
supports multiple API calls like Javascript, Python, SQL...
44 / 110

2. CRUD Operations
based on Google Protocol Buffers
45 / 110

2. CRUD Operations
based on Google Protocol Buffers
4. MySQL Shell
46 / 110

MySQL Document Store
47 / 110

MySQL as Document Store
Starting with DS
48 / 110

Starting using MySQL DS in few minutes
Install X Plugin
49 / 110

Starting using MySQL DS in few minutes

50 / 110

Migration from MongoDB to MySQL DS
For this example, I will use the well known restaurants collection:
51 / 110

We need to dump the data to a file and we will use MySQL Shell with the Python
interpreter to load the data.
52 / 110

We need to dump the data to a file and we will use MySQL Shell with the Python
interpreter to load the data.
53 / 110

Dump and load using MySQL Shell & Python
this example is inspired by @datacharmer's work:
https://www.slideshare.net/datacharmer/mysql-documentstore
$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000;
db.restaurants. nd().shellPrint()'
| perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/ $1/g' > all_recs.json
54 / 110

Dump and load using MySQL Shell & Python
this example is inspired by @datacharmer's work:
https://www.slideshare.net/datacharmer/mysql-documentstore
$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000;
db.restaurants. nd().shellPrint()'
| perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/ $1/g' > all_recs.json
55 / 110

56 / 110

Let´s query
57 / 110

Let´s query
That´s too much records to show it here... let´s limit it
58 / 110

59 / 110

Some more examples
60 / 110

Some more examples
Let´s add a selection criteria:
61 / 110

Syntax slightly different than MongoDB
62 / 110

Syntax slightly different than MongoDB
> db.restaurants. nd({"cuisine": "French",
"borough": { $not: /^Manhattan/} },
{"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2)
{ "borough" : "Queens", "cuisine" : "French",
"name" : "La Baraka Restaurant" }
{ "borough" : "Queens", "cuisine" : "French",
"name" : "Air France Lounge" }
63 / 110

CRUD operations
64 / 110

8.0.3 what´s new ?
WL10446 - Customizable prompts
WL10568 - Handling of Ctrl+C has been improved on many scenarios
WL10708 and WL10823 - NodeSession disappeared in favor of simply Session
WL10912 - Changes on connection options handling:
case insensitive: i.e. ssl-mode == Ssl-Mode
renamed from camel case to word dashed format: sslMode --> ssl-mode
WL11054 - Admin API supports automatic turn of of super-read-only option on
several functions.
WL10682 - Mysqlx.CRUD.Update on top level document
65 / 110

8.0.3 what´s new ? (part 2)
WL10645 - X Protocol Crud.Find with row locking
WL10734 - Spatial index type
WL10744 - Full-Text Index Type
WL10797 - Mysqlx.Crud.Update: MERGE_PATCH
WL10447 - auto-completion by hitting in the middle of typing a shell command,
database object, object function, ...
66 / 110

8.0.3 labs: auto-completion
67 / 110

8.0.3 labs: auto-completion
68 / 110

8.0.3 labs: auto-completion - even for SQL !
69 / 110

what about my old SQL ?
The hidden part of the iceberg
70 / 110

JSON datatype is behind the scene
native datatype (since 5.7.8)
71 / 110

JSON values are stored in MySQL tables using UTF8MB4
72 / 110

Conversion from "native" SQL types to JSON values
73 / 110

JSON manipulation functions (JSON_EXTRACT, JSON_KEYS,
JSON_SEARCH,...)
74 / 110

JSON_SEARCH,...)
Generated/virtual columns
75 / 110

JSON_SEARCH,...)
Indexing JSON data
76 / 110

JSON_SEARCH,...)
Indexing JSON data
Foreign Keys to JSON data
77 / 110

JSON_SEARCH,...)
Indexing JSON data
Foreign Keys to JSON data
SQL Views to JSON data
78 / 110

How does it work ?
79 / 110

Mapping to SQL examples
createCollection('mycollection')
80 / 110

CREATE TABLE `test`.`mycoll` (
doc JSON,
_id VARCHAR(32)
GENERATED ALWAYS AS (doc->>'$._id') STORED
PRIMARY KEY
) CHARSET utf8mb4;
81 / 110

doc JSON,
_id VARCHAR(32)
PRIMARY KEY
) CHARSET utf8mb4;
mycollection.add({'test': 1234})
82 / 110

doc JSON,
_id VARCHAR(32)
PRIMARY KEY
) CHARSET utf8mb4;
mycollection.add({'test': 1234})
INSERT INTO `test`.`mycoll` (doc)
VALUES (
JSON_OBJECT('_id','663807fe367ee6114e0e5458bdac28bf',
'test',1234)
);
83 / 110

Mapping to SQL examples (2)
mycollection.find("test > 100")
84 / 110

Mapping to SQL examples (2)
mycollection.find("test > 100")
SELECT doc
FROM `test`.`mycoll`
WHERE (JSON_EXTRACT(doc,'$.test') > 100);
85 / 110

SQL and JSON Example
86 / 110

same as:
87 / 110

same as:
we can use it in SQL:
88 / 110

SQL and JSON Example (2): validation
89 / 110

90 / 110

91 / 110

SQL and JSON Example (3): explain
92 / 110

SQL and JSON Example (3): explain
93 / 110

SQL and JSON Example (4): add index
94 / 110

SQL and JSON Example (4): add index
95 / 110

SQL and JSON Example (5): arrays
96 / 110

97 / 110

98 / 110

99 / 110

NoSQL as SQL
New in MySQL 8.0.3-labs: JSON_TABLE
100 / 110

NoSQL as SQL (2)
New in MySQL 8.0.3-labs: JSON_TABLE
101 / 110

SQL and NoSQL
102 / 110

SQL and NoSQL
103 / 110

SQL and NoSQL
104 / 110

SQL and NoSQL (2)
105 / 110

SQL and NoSQL (2)
106 / 110

SQL and NoSQL (2)
107 / 110

what do I gain ?
Conclusion
108 / 110

schemaless
flexible data structure
easy to start (CRUD)
Conclusion
This is the best of the two worlds in one product !
Data integrity
ACID Compliant
SQL
109 / 110

Thank you !
Any Questions ?
http://lefred.be
110 / 110

MySQL Document Store - How to replace a NoSQL database by MySQL without effort but with a lot gains ?

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MySQL Document Store - How to replace a NoSQL database by MySQL without effort but with a lot gains ?

Similar to MySQL Document Store - How to replace a NoSQL database by MySQL without effort but with a lot gains ? (20)

More from Frederic Descamps

More from Frederic Descamps (20)

Recently uploaded

Recently uploaded (20)

MySQL Document Store - How to replace a NoSQL database by MySQL without effort but with a lot gains ?