20171104 hk-py con-mysql-documentstore_v1

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Using MySQL Shell
High Availability + Document Store
with Python
Cross(X)over between NoSQL and SQL
Ivan Ma
2017-11-04

Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2

Program Agenda
MySQL Document Store Introduction
Components of the MySQL Document Store
The X DevAPI – A modern CRUD App Programming Interface
Documents via SQL
Behind the Scenes – CRUD to SQL Translation
1
2
3
4
5
3

MySQL Document Store
Introduction
4

Document Oriented Databases
• Representing complex information, similar to an Object
• JSON (=JavaScript Object Notation)
– Compact, popular and standardized
– Can be represented natively in many languages (JavaScript,
Python etc)
• Other popular encoding formats are XML, YAML etc
What is a Document?
5
{
"_id": "AUT",
"Name": "Austria",
"GNP": 211860,
"IndepYear": 1918,
"demographics": {
"LifeExpectancy": 77.699,
"Population": 8091800
},
"geography": {
"Continent": "Europe",
"Region": "Western Europe",
"SurfaceArea": 83859
}
}
JSON Document Example

Relational Databases vs Document Store Systems
Relational Model Document Model - JSON
6
id name email city_id
3412 John Smith john@oracle.com 45
id city country_id
45 San Francisco US
customer
city
{
_id: 3412,
name: "John Smith",
email: john@oracle.com,
city: "San Francisco",
country: "US",
orders: [
{date: "2017-08-24", total: 312.20},
{date: "2017-10-02", total: 24.95}
]
}
id id_customer date total
381 3412 2017-08-24 312.20
412 3412 2017-10-02 24.95
shop_order

The MySQL Document Store
• Document oriented data storage for MySQL
– Enables working with both, structured and unstructured data
– Exposes new CRUD interface, while keeping SQL
• Standard plugin for MySQL Server (since MySQL 5.7.12)
– Today (2017-10-16 – MySQL 5.7.20 Released)
– MySQL 8.0.3 RC
• Store JSON documents in MySQL
– Through traditional SQL interface
or
– New X DevAPI NoSQL interface (much easier to work with for documents)
7

Components of the
8

API / Client
Administration
Server Side
9
MySQL Document Store - Components
MySQL Connectors
MySQL Shell
X Plugin
X DevAPI
X Protocol
MySQL Server
JSON Datatype
New JSON Syntax
New JSON Functions

• MySQL Server 5.7 or 8.0
• JSON Datatype
• MySQL X Plugin
• Introduces X Protocol for relational- and
document operations
• Maps CRUD operations to standard SQL
(relational tables, JSON datatype and
functions)
• X Protocol
• New MySQL client protocol based on top of
industry standard (Protobuf)
• Works for both, CRUD and SQL operations
• X DevAPI
• New, modern, async developer API for CRUD
and SQL operations on top of X Protocol
• Introduces Collections as new Schema obj.
• MySQL Shell
• Offers interactive X DevAPI mode for app
prototyping
• MySQL Connectors
• Support for X DevAPI and X Session
10
MySQL Document Store – Components

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 11
X Plugin
33060
MySQL Shell
X Protocol
JavaScript / Python
Node.js Application
MySQL Connector/Node.js
...
X DevAPI
X DevAPI
Windows Application
MySQL Connector/Net X DevAPI
MySQL Server
MySQL Server MySQL Server
MySQL Router
MySQL Router
X Plugin
33060
X Plugin
33060
MySQL
InnoDB
cluster

MySQL Document Store running on InnoDB Cluster
• Document Store is tightly connected to InnoDB Cluster
– Main goals: Ease-of-use & Scalability
• Document Store opens an easier path to massive write scale-out
– Sharding with the relational model is hard
– Self contained documents can be sharded much easier
12

MySQL Connector
Application
MySQL Connector
Application
MySQL Shell
MySQL Connector
Application
MySQL Connector
Application
Running Doc Store on MySQL InnoDB Cluster
MySQL
InnoDB
cluster
MySQL Enterprise Monitor
…

Live Demo

The X DevAPI –
A modern CRUD App Programming Interface
15

MySQL Document Store - X DevAPI
• CRUD stands for the following 4 basic operations
– CREATE
– READ
– UPDATE
– DELETE
CRUD
Confidential – Oracle
16

Examples
• session.get_schemas()
• mydb = session.get_schema(‘world_x’)
• mydb.get_collections()
17

Examples
• session.get_schemas()
• mydb = session.get_schema(‘world_x’)
• mydb.get_collections()
• mydb.help()
18

More examples with DOC operation with MySQL Shell
• mydb.countryinfo.find()
• mydb.create_collectin(‘mytest’);
• mydb.mytest.add( …..)
• mydb.mytest.find()
19

• The Collection object represents a collection of JSON Documents
Example: python
products = db.create_collection('products');
Collections
20

• Add is used to add new JSON Documents to a Collection
Example:
products.add({"_id": 123, "name": "Vacuum Cleaner", "model": "UH20040", "brand":
"Hoover", "price": 79.99}).execute();
products.add({"_id": 124, "name": "Ninja Master Prep", "brand": "Ninja", "model": "QB900B",
"price": 34.88, "material": "Plastic", "color": "Silver"}).execute();
CRUD: Add
21

• Find is used to retrieve JSON Documents from a collection
Example:
products.find("price > 10 and price < 100").fields([”_id", "name", "price"]).sort(["name"]).execute();
CRUD: Find
22

• Modify is used to update existing JSON Documents of a Collection
Example:
products.modify(”_id = 123").set("dimensions.weight", 50).execute();
CRUD: Modify
23

• Remove is used to delete JSON Documents from a Collection
Example:
products.remove("_id = 123");
CRUD: Remove
24

• JavaScript / Node.js – Connector/Node.js 8.0.8-dmr
• Get it from npm or https://dev.mysql.com/downloads/connector/nodejs/
• API Reference: https://dev.mysql.com/doc/dev/connector-nodejs/
• C#/.NET – Connector/Net 8.0.9-dmr
• Get it from NuGet or http://dev.mysql.com/downloads/connector/net/
• API Reference: http://dev.mysql.com/doc/dev/connector-net
• Java – Connector/J 8.0.8-dmr
• Get it from http://dev.mysql.com/downloads/connector/j/
• API Reference: https://dev.mysql.com/doc/dev/connector-j/
25
Supported Languages

• Python – Connector/Python 8.0.5-dmr
• Get it from http://dev.mysql.com/downloads/connector/python/
• API Reference: https://dev.mysql.com/doc/dev/connector-python/
• C++ – Connector/C++ 8.0.6-dmr
• Get it from https://dev.mysql.com/downloads/connector/cpp/
• API Reference: https://dev.mysql.com/doc/dev/connector-cpp/devapi_ref.html
• PHP – MySQL native driver for PHP (soon)
26
Supported Languages

• X DevAPI Users Guide
– https://dev.mysql.com/doc/x-devapi-userguide/en/
• Using MySQL as a Document Store
– https://dev.mysql.com/doc/refman/5.7/en/document-store.html
27
More information

Get ready to try the
28

Getting Ready
• Fast application prototyping using MySQL Shell
– We will write a simple Document Store application in the MySQL Shell
– We will create MySQL InnoDB Cluster of 3 nodes
• Python creating MySQL InnoDB Cluster
• Python to create document store on MySQL InnoDB Cluster
– Accessing Document using CRUD
– Accessing Document using SQL
What is the goal of the demo?
29

Live Demo

Documents via SQL
31

Document Storage with SQL
• Allows storage of JSON data in a MySQL table column
• Validations on INSERT/UPDATE
• Internal binary storage format allows fast key lookups inside documents
• Use inline JSON in SQL expressions (compare JSON values etc.)
CREATE TABLE mytable (
id INT PRIMARY KEY,
data JSON,
…);
JSON Datatype
32

• JSON data manipulation in SQL
• Construct JSON values (JSON_OBJECT(), JSON_ARRAY())
• Extract information from JSON objects and arrays (JSON_EXTRACT())
• Modify JSON values (JSON_SET(), JSON_INSERT() etc.)
• Inline JSON Path Syntax to reference values inside JSON data in SQL
– [[schema.]table.]column->'$.field.array[0]'
JSON Functions
33

• Virtual columns allow indexes on JSON fields
– Create a virtual column to "look into" a JSON document
– Create index on the virtual column
• Foreign keys can also be created on virtual columns
Indexing JSON Data
34

id name email city_id
3412 John Smith john@oracle.com 45
Use Case: Relational to JSON Data
35
id city country_id
45 San Francisco US
{
id: 3412,
name: "John Smith",
email: john@oracle.com,
city: "San Francisco",
country: "US"
}
customer
city

Use Case: Relational to JSON Data
36
SELECT JSON_OBJECT('id', cu.id,
'name', cu.name,
'email', cu.email,
'city', ci.city) as customer
FROM customer cu
JOIN city ci ON ci.id = cu.city_id

Use Case: Query JSON Data via SQL
37
SELECT doc->'$.id' as id,
doc->'$.name' as name,
doc->'$.email' as email
FROM customer
WHERE doc->'$.city' = 'San Francisco'

Behind the Scenes
38

How it Works
• Applications use DevAPI connectors to write database operations in native
code (instead of SQL)
• Connector translates DevAPI operations to X protocol document requests
• X Plugin translates document requests to SQL
• Results sent back to application as JSON documents
39

MySQL Document Store – How it works
• Existing applications are not touched
– Developers adopt DocStore only if they want to
• X Protocol allows SQL and CRUD
– SQL: SELECT * FROM myTable WHERE id = 12;
– X DevAPI: db.myTable.find("id=12");
• Directly hooked into the optimizer
– Bypass MySQL connection handling
Storage
Optimizer
MySQL
Connection
handling
X Plugin
MySQL Server 5.7 & 8.0
330633060
MySQL Shell
Existing Application
MySQL Connector/ODBC
SQL
Result
Classic MySQL ProtocolX Protocol
JavaScript / Python
Node.js Application
MySQL C/Node.js
...
X DevAPI
X DevAPI
Windows Application
MySQL Connector/Net X DevAPI

How it Works: Mapping to SQL
CREATE TABLE `test`.`mycoll` (
doc JSON,
_id VARCHAR(32)
GENERATED ALWAYS AS (doc->>'$._id') STORED
PRIMARY KEY
) CHARSET utf8mb4;
mycollection = create_collection('mycollection')
41

INSERT INTO `test`.`mycoll` (doc)
VALUES (
JSON_OBJECT('_id','663807fe367ee6114e0e5458bdac28bf',
'test',1234)
);
mycollection.add({'test': 1234})
42

SELECT doc
FROM `test`.`mycoll`
WHERE (JSON_EXTRACT(doc,'$.test') > 100);
mycollection.find("test > 100")
43

Summary and Take Away
44
• New, modern way to develop database applications
• Combine best of relational and document oriented models
• MySQL InnoDB Cluster – Future proof for HA and scale-out deployments
• Blogs: mysqlserverteam.com/category/docstore/
• https://dev.mysql.com/doc/refman/5.7/en/mysql-shell-tutorial-python.html

Thank you!
45

20171104 hk-py con-mysql-documentstore_v1

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to 20171104 hk-py con-mysql-documentstore_v1

Similar to 20171104 hk-py con-mysql-documentstore_v1 (20)

More from Ivan Ma

More from Ivan Ma (14)

Recently uploaded

Recently uploaded (20)

20171104 hk-py con-mysql-documentstore_v1