This document provides an overview of different ways to work with MySQL using standard SQL, X DevAPI, and MySQL Shell utilities. It discusses querying, updating, and exporting/importing data using these different approaches. It also covers topics like character encoding, generating summaries, storing errors, and retrieving metadata. Examples are provided to illustrate concepts like selecting, grouping, joining, changing data, common table expressions, and more using SQL and X DevAPI. MySQL Shell utilities for exporting/importing CSV, JSON, and working with collections are also demonstrated.
2. l
Sveta Smirnova
• MySQL Support Engineer
• Author
MySQL Troubleshooting
MySQL Cookbook, 4th Edition
• JSON UDF functions
• FILTER clause for MySQL
• Speaker
• Percona Live, OOW, Fosdem,
DevConf, HighLoad...
3. Sveta the Speaker
• Audience: DBAs
• They do not write queries
• They tune
• Server options
• Indexes
• Table structure
3
4. MySQL Cookbook by O’Reilly
• For developers
• New topic for me → challenging
• In past I worked as a developer
4
5. MySQL Cookbook by O’Reilly
• For developers
• New topic for me → challenging
• In past I worked as a developer
• I accepted
5
6. 15 years ago
• Queries
sql = "SELECT name, lastname " +
"FROM my_table " +
"WHERE id = " + my_id
6
7. 15 years ago
• Placeholders
sql = "SELECT name, lastname " +
"FROM my_table " +
"WHERE id = ?"
7
8. 15 years ago
• ORM
Speaker(name="Sveta", lastname="Smirnova").save()
8
12. MySQL CLI
mysql > SELECT ’Hello, world!’;
+---------------+
| Hello, world! |
+---------------+
| Hello, world! |
+---------------+
1 row in set (0,00 sec)
12
13. MySQL CLI
• Your SQL code debugger
• Tested by
• Millions of users
• MySQL developers
• Hundreds of tests at every release
• Model API for your query
13
14. MySQL Shell
MySQL JS > print("Hello, world!")
Hello, world!
MySQL JS > py
Switching to Python mode...
MySQL Py > print("Hello, world!")
Hello, world!
MySQL Py > sql
Switching to SQL mode... Commands end with ;
MySQL SQL > SELECT ’Hello, world!’;
+---------------+
| Hello, world! |
+---------------+
| Hello, world! |
+---------------+
1 row in set (0.0003 sec)
14
15. MySQL Shell
• New command-line client with X DevAPI support
• SQL and Object-Oriented queries
• MySQL Server administration
• Utilities
• Replication
• InnoDB Cluster
• Your own applications
15
16. X DevAPI
• Asynchronous code execution
• Works with MySQL
• As usual: by executing SQL
• Querying tables as documents
• Collections and documents support
Data storage in JSON
NoSQL-syntax, similar to MongoDB’s
16
17. Reading
Standard SQL
SQL > SELECT thing, SUM(legs+arms) AS limbs
-> FROM limbs GROUP BY thing
-> HAVING limbs > 5;
+-----------+-------+
| thing | limbs |
+-----------+-------+
| armchair | 6 |
| centipede | 99 |
| insect | 6 |
| squid | 10 |
+-----------+-------+
4 rows in set (0.0004 sec)
17
21. Working with Results
SQL code in Python
cursor = conn.cursor()
cursor.execute("SELECT id, name, cats FROM profile")
while True:
row = cursor.fetchone()
if row is None:
break
print(f"id: row[0], name: row[1], cats: row[2]")
21
22. Working with Results
X DevAPI for tables
result = session.get_schema(’cookbook’).
get_table(’profile’).
select(’id’, ’name’, ’cats’).execute()
while True:
row = result.fetch_one()
if row is None:
break
print(f"id: row[0], name: row[1], cats: row[2]")
22
23. Working with Results
Document Store
result = session.get_schema(’cookbook’).
get_collection(’collectionProfile’).
find().execute()
while True:
doc = result.fetch_one()
if doc is None:
break
print(f"id: doc[’id’], name: doc[’name’], cats: doc[’cats’]")
23
24. Working with Results
Tables and Document Store
result = session.get_schema(’cookbook’).
get_collection_as_table(’collectionProfile’).
select(’JSON_EXTRACT(doc, "$.id") AS id’).
select(’JSON_EXTRACT(doc, "$.name") AS name’).
select(’JSON_EXTRACT(doc, "$.cats") AS cats’).execute()
while True:
row = result.fetch_one()
if row is None:
break
print(f"id: row[0], name: row[1], cats: row[2]")
24
32. Generating Summaries
• When was the longest trip per driver?
+--------+-------+------------+-------+
| rec_id | name | trav_date | miles |
+--------+-------+------------+-------+
| 1 | Ben | 2014-07-30 | 152 |
| 2 | Suzi | 2014-07-29 | 391 |
| 3 | Henry | 2014-07-29 | 300 |
| 4 | Henry | 2014-07-27 | 96 |
| 5 | Ben | 2014-07-29 | 131 |
| 6 | Henry | 2014-07-26 | 115 |
| 7 | Suzi | 2014-08-02 | 502 |
| 8 | Henry | 2014-08-01 | 197 |
| 9 | Ben | 2014-08-02 | 79 |
| 10 | Henry | 2014-07-30 | 203 |
+--------+-------+------------+-------+
32
33. Generating Summaries
Naive solution
mysql> SELECT name, trav_date, MAX(miles) AS ’longest trip’
-> FROM driver_log GROUP BY name;
ERROR 1055 (42000): ’cookbook.driver_log.trav_date’ isn’t in GROUP BY
mysql> SET sql_mode=”;
Query OK, 0 rows affected (0,00 sec)
mysq> SELECT name, trav_date, MAX(miles) AS ’longest trip’
-> FROM driver_log GROUP BY name;
+-------+------------+--------------+
| name | trav_date | longest trip |
+-------+------------+--------------+
| Ben | 2014-07-30 | 152 |
| Suzi | 2014-07-29 | 502 |
| Henry | 2014-07-29 | 300 |
+-------+------------+--------------+
3 rows in set (0,00 sec)
33
34. Generating Summaries
Legacy solution
mysql> CREATE TEMPORARY TABLE t
-> SELECT name, MAX(miles) AS miles
-> FROM driver_log GROUP BY name;
mysql> SELECT d.name, d.trav_date, d.miles AS ’longest trip’
-> FROM driver_log AS d INNER JOIN t USING (name, miles)
-> ORDER BY name;
+-------+------------+--------------+
| name | trav_date | longest trip |
+-------+------------+--------------+
| Ben | 2014-07-30 | 152 |
| Henry | 2014-07-29 | 300 |
| Suzi | 2014-08-02 | 502 |
+-------+------------+--------------+
mysql> DROP TABLE t;
34
35. Generating Summaries
Common Table Expression (CTE)
mysql> WITH t AS
-> (SELECT name, MAX(miles) AS miles
-> FROM driver_log GROUP BY name)
-> SELECT d.name, d.trav_date, d.miles AS ’longest trip’
-> FROM driver_log AS d INNER JOIN t USING (name, miles)
-> ORDER BY name;
+-------+------------+--------------+
| name | trav_date | longest trip |
+-------+------------+--------------+
| Ben | 2014-07-30 | 152 |
| Henry | 2014-07-29 | 300 |
| Suzi | 2014-08-02 | 502 |
+-------+------------+--------------+
3 rows in set (0.01 sec)
35
36. Storing Errors with Diagnostic Area
Foreign keys
ALTER TABLE movies_actors_link ADD FOREIGN KEY(movie_id) REFERENCES movies(id);
ALTER TABLE movies_actors_link ADD FOREIGN KEY(actor_id) REFERENCES actors(id);
36
37. Storing Errors with Diagnostic Area
Custom error log
CREATE TABLE ‘movies_actors_log‘ (
‘err_ts‘ timestamp NULL DEFAULT CURRENT_TIMESTAMP
ON UPDATE CURRENT_TIMESTAMP,
‘err_number‘ int DEFAULT NULL,
‘err_sqlstate‘ char(5) DEFAULT NULL,
‘err_message‘ TEXT DEFAULT NULL,
‘movie_id‘ int unsigned DEFAULT NULL,
‘actor_id‘ int unsigned DEFAULT NULL
);
37
38. Storing Errors with Diagnostic Area
Data modification procedure
CREATE PROCEDURE insert_movies_actors_link(movie INT, actor INT)
BEGIN
DECLARE e_number INT;
DECLARE e_sqlstate CHAR(5);
DECLARE e_message TEXT;
DECLARE CONTINUE HANDLER FOR SQLEXCEPTION
BEGIN
GET DIAGNOSTICS CONDITION 1
e_number = MYSQL_ERRNO, e_sqlstate = RETURNED_SQLSTATE, e_message = MESSAGE_TEXT;
INSERT INTO movies_actors_log(err_number, err_sqlstate,
err_message, movie_id, actor_id)
VALUES(e_number, e_sqlstate, e_message, movie, actor);
RESIGNAL;
END;
INSERT INTO movies_actors_link VALUES(movie, actor);
END
38
39. Storing Errors with Diagnostic Area
After few errors
mysql> SELECT err_ts, err_message, movie_id, actor_id
-> FROM movies_actors_logG
*************************** 1. row ***************************
err_ts: 2022-02-28 04:23:28
err_message: Cannot add or update a child row: a foreign key
constraint fails (‘cookbook‘.‘movies_actors_link‘,
CONSTRAINT ‘movies_actors_link_ibfk_1‘
FOREIGN KEY (‘movie_id‘) REFERENCES ‘movies‘ (‘id‘))
movie_id: 7
actor_id: 11
...
39
40. Storing Errors with Diagnostic Area
After few errors
*************************** 2. row ***************************
err_ts: 2022-02-28 04:23:38
err_message: Column ’movie_id’ cannot be null
movie_id: NULL
actor_id: 10
*************************** 3. row ***************************
err_ts: 2022-02-28 04:23:48
err_message: Duplicate entry ’6-9’ for key
’movies_actors_link.movie_id’
movie_id: 6
actor_id: 9
40
42. Export and Import
• mysqlpump: mysqldump on steroids
• Legacy commands removed
ALTER TABLE ... DISABLE KEYS
• Multithreaded
--default-parallelism
• More filters
--exclude-*
--include-*
--skip-definer
42
43. Export and Import
• MySQL Shell Utilities
Export into CSV
JS > util.exportTable(
-> ’limbs’, ’BACKUP/cookbook/limbs.csv’,
-> dialect: "csv-unix")
Import from CSV
Py > sql CREATE TABLE test.limbs LIKE limbs;
Fetching table and column names from ‘cookbook‘
for auto-completion...
Press ^C to stop.
Query OK, 0 rows affected (0.0264 sec)
Py > util.import_table("BACKUP/cookbook/limbs.csv",
"dialect": "csv-unix", "schema": "test")
43
44. Export and Import
• MySQL Shell Utilities
• Import from JSON
$ mysqlsh cbuser:cbpass@127.0.0.1:33060/cookbook
> --import limbs.json CollectionLimbs
WARNING: Using a password on the command line interface
can be insecure.
Importing from file "limbs.json" to collection
‘cookbook‘.‘CollectionLimbs‘ in MySQL Server
at 127.0.0.1:33060
.. 11.. 11
Processed 506 bytes in 11 documents in 0.0067 sec
(11.00 documents/s)
Total successfully imported documents 11
(11.00 documents/s)
44
45. Export and Import
• MySQL Shell Utilities
Import from mongoexport
JS > options = {
-> "schema": "cookbook",
-> "collection": "blogs",
-> "convertBsonTypes": true
-> }
->
JS > util.importJson("blogs.json", options)
Importing from file "blogs.json" to collection
‘cookbook‘.‘blogs‘ in MySQL Server at 127.0.0.1:33060
.. 2.. 2
Processed 240 bytes in 2 documents in 0.0070 sec
(2.00 documents/s)
Total successfully imported documents 2 (2.00 documents/s)
45
46. Export and Import
• MySQL Shell Utilities
• Export into JSON
$ mysqlsh cbuser:cbpass@127.0.0.1:33060/cookbook
> -e "limbs=shell.getSession().getSchema(’cookbook’).
> getCollection(’collectionLimbs’).
> find().execute().fetchAll();
> println(limbs);" > limbs.json
46
47. Metadata
Column metadata
def get_enumorset_info(client, db_name, tbl_name, col_name)
sth = client.prepare(
"SELECT COLUMN_NAME, COLUMN_TYPE, IS_NULLABLE, COLUMN_DEFAULT
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ? AND COLUMN_NAME = ?")
res = sth.execute(db_name, tbl_name, col_name)
return nil if res.count == 0 # no such column
row = res.first
info = {}
info["name"] = row.values[0]
return nil unless row.values[1] =~ /^(ENUM|SET)((.*))$/i
info["type"] = $1
info["values"] = $2.split(",").collect |val| val.sub(/^’(.*)’$/, "1")
info["nullable"] = (row.values[2].upcase == "YES")
info["default"] = row.values[3]
return info
end
47
48. Metadata
Checking in the application
info = get_enumorset_info(conn, db_name, tbl_name, col_name)
if info is not None and info[’type’].upper() == ’ENUM’:
valid = 1 if list(map(lambda v: v.upper(), info[’values’])).count(val.upper()) > 0 else 0
48
49. Metadata
All tables, referenced by foreign keys
mysql> SELECT ku.TABLE_NAME AS child,
-> ku.REFERENCED_TABLE_NAME AS parent
-> FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS tc
-> JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE ku
-> USING (CONSTRAINT_NAME, TABLE_SCHEMA, TABLE_NAME)
-> WHERE CONSTRAINT_TYPE=’FOREIGN KEY’ AND
-> ku.TABLE_SCHEMA=’cookbook’;
+--------------------+--------+
| child | parent |
+--------------------+--------+
| movies_actors_link | movies |
| movies_actors_link | actors |
+--------------------+--------+
2 rows in set (0,00 sec)
49
50. Metadata
Result: standard API
cursor = conn.cursor(buffered=True)
cursor.execute(stmt)
# metadata information becomes available at this point ...
print("Number of rows: %d" % cursor.rowcount)
if cursor.with_rows:
ncols = len(cursor.description)
else:
ncols = 0 # no result set
print("Number of columns: %d" % ncols)
if ncols == 0:
print("Note: statement has no result set")
else:
for i, col_info in enumerate(cursor.description):
name, type, _, _, _, _, nullable, flags, *_ = col_info
print("--- Column %d (%s) ---" % (i, name))
...
50
51. Metadata
• Result: X DevAPI
• Only basic information
"executionTime": "0.0007 sec",
"affectedRowCount": 0,
"affectedItemsCount": 0,
"warningCount": 0,
"warningsCount": 0,
"warnings": [],
"info": "",
"autoIncrementValue": 0
51
52. Validation and Formatting
CHECK Constraints
mysql> ALTER TABLE patients ADD CONSTRAINT date_check
-> CHECK((date_departed IS NULL) OR
-> (date_departed >= date_arrived));
mysql> INSERT INTO patients (national_id, name, surname,
-> gender, age, diagnosis, date_arrived, date_departed)
-> VALUES(’34GD429520’, ’John’, ’Doe’, ’M’, 45,
-> ’Data Phobia’, ’2020-07-20’, ’2020-05-31’);
ERROR 3819 (HY000): Check constraint ’date_check’ is violated.
52
55. Sequences
Enumerating result
mysql> SELECT
-> ROW_NUMBER() OVER win AS turn,
-> first_name, last_name FROM name
-> WINDOW win
-> AS (ORDER BY RAND());
+------+------------+-----------+
| turn | first_name | last_name |
+------+------------+-----------+
| 1 | Devon | White |
| 2 | Kevin | Brown |
| 3 | Rondell | White |
| 4 | Vida | Blue |
| 5 | Pete | Gray |
+------+------------+-----------+
5 rows in set (0.00 sec)
55
56. Sequences
Several sequences in one query
mysql> WITH RECURSIVE sequences(id, geo, random) AS
-> (SELECT 1, 3, FLOOR(1+RAND()*5)
-> UNION ALL
-> SELECT id + 1, geo * 4, FLOOR(1+RAND()*5) FROM sequences WHERE id < 5)
-> SELECT * FROM sequences;
+------+------+--------+
| id | geo | random |
+------+------+--------+
| 1 | 3 | 4 |
| 2 | 12 | 4 |
| 3 | 48 | 2 |
| 4 | 192 | 2 |
| 5 | 768 | 3 |
+------+------+--------+
5 rows in set (0.00 sec)
56
57. Joins and Subqueries
How many drivers on the road?
mysql> SELECT trav_date, COUNT(trav_date) AS drivers
-> FROM driver_log GROUP BY trav_date ORDER BY trav_date;
+------------+---------+
| trav_date | drivers |
+------------+---------+
| 2014-07-26 | 1 |
| 2014-07-27 | 1 |
| 2014-07-29 | 3 |
| 2014-07-30 | 2 |
| 2014-08-01 | 1 |
| 2014-08-02 | 2 |
+------------+---------+
57
59. Joins and Subqueries
When drivers have rest?
mysql> SELECT dates.d
-> FROM dates LEFT JOIN driver_log
-> ON dates.d = driver_log.trav_date
-> WHERE driver_log.trav_date IS NULL;
+------------+
| d |
+------------+
| 2014-07-28 |
| 2014-07-31 |
+------------+
59
60. Joins and Subqueries
CTE: in one query
WITH RECURSIVE dates (d) AS (
SELECT ’2014-07-26’
UNION ALL
SELECT d + INTERVAL 1 day
FROM dates
WHERE d < ’2014-08-02’)
SELECT dates.d, COUNT(driver_log.trav_date) AS drivers
FROM dates LEFT JOIN driver_log
ON dates.d = driver_log.trav_date
GROUP BY d ORDER BY d;
60
62. Statistics
The issue!
mysql> SHOW WARNINGSG
*************************** 1. row ***************************
Level: Warning
Code: 1287
Message: Setting user variables within expressions is
deprecated and will be removed in a future release. Consider
alternatives: ’SET variable=expression, ...’, or
’SELECT expression(s) INTO variables(s)’.
1 row in set (0,00 sec)
62
65. Duplicates
Duplicate users
mysql> WITH tmp AS (
-> SELECT COUNT(*) AS count, last_name, first_name
-> FROM catalog_list GROUP BY last_name, first_name HAVING count > 1)
-> SELECT catalog_list.*
-> FROM tmp INNER JOIN catalog_list USING (last_name, first_name)
-> ORDER BY last_name, first_name;
+-----------+------------+----------------------+
| last_name | first_name | street |
+-----------+------------+----------------------+
| Baxter | Wallace | 57 3rd Ave. |
| BAXTER | WALLACE | 57 3rd Ave. |
| Baxter | Wallace | 57 3rd Ave., Apt 102 |
| Pinter | Marlene | 9 Sunset Trail |
| Pinter | Marlene | 9 Sunset Trail |
+-----------+------------+----------------------+
5 rows in set (0,00 sec)
65
66. JSON
• Data type JSON
• Compact storage
• In-place update
On the source and replica
binlog_row_value_options=PARTIAL_JSON
• Operators -> and ->>
• Functions
Search, pattern support
Update
Validation, including JSON schema
Conversion
Documents join
66
67. JSON
• JSON Path for queries
Names of book authors
mysql> SELECT JSON_EXTRACT(author, ’$.name’) AS author
-> FROM book_authors;
+---------+
| author |
+---------+
| "Paul" |
| "Alkin" |
| "Sveta" |
+---------+
3 rows in set (0,00 sec)
67
68. JSON
• JSON Path for queries
Names of book authors
mysql> SELECT author->’$.name’ AS author
-> FROM book_authors;
+---------+
| author |
+---------+
| "Paul" |
| "Alkin" |
| "Sveta" |
+---------+
3 rows in set (0,00 sec)
68
69. JSON
• JSON Path for queries
Removing quotes
mysql> SELECT JSON_UNQUOTE(
-> JSON_EXTRACT(author, ’$.name’)
-> ) AS author FROM book_authors;
+--------+
| author |
+--------+
| Paul |
| Alkin |
| Sveta |
+--------+
3 rows in set (0,00 sec)
69
70. JSON
• JSON Path for queries
Removing quotes
mysql> SELECT author-»’$.name’ AS author
-> FROM book_authors;
+--------+
| author |
+--------+
| Paul |
| Alkin |
| Sveta |
+--------+
3 rows in set (0,00 sec)
70
71. JSON
• JSON Path for queries
First and last books
mysql> SELECT CONCAT(author-»’$.name’, ’ ’, author-»’$.lastname’) AS author,
-> author-»’$.books[0]’ AS ‘First Book‘, author-»’$.books[last]’ AS ‘Last Book‘
-> FROM book_authorsG
************************ 1. row ************************
author: Paul DuBois
First Book: Software Portability with imake: ...
Last Book: MySQL (Developer’s Library)
************************ 2. row ************************
author: Alkin Tezuysal
First Book: MySQL Cookbook
Last Book: MySQL Cookbook
************************ 3. row ************************
author: Sveta Smirnova
First Book: MySQL Troubleshooting
Last Book: MySQL Cookbook
71
72. JSON
Indexes
mysql> ALTER TABLE book_authors
-> ADD COLUMN lastname VARCHAR(255)
-> GENERATED ALWAYS AS
-> (JSON_UNQUOTE(JSON_EXTRACT(author, ’$.lastname’)));
mysql> ALTER TABLE book_authors
-> ADD COLUMN name VARCHAR(255)
-> GENERATED ALWAYS AS (author-»’$.name’);
mysql> CREATE INDEX author_name
-> ON book_authors(lastname, name);
72
73. Transactions
Standard API
conn.start_transaction()
cursor = conn.cursor()
# move some money from one person to the other
cursor.execute("UPDATE money SET amt = amt - 6
WHERE name = ’Eve’")
cursor.execute("UPDATE money SET amt = amt + 6
WHERE name = ’Ida’")
cursor.close()
conn.commit()
73
74. Transactions
X DevAPI for tables
session.get_schema(’cookbook’).get_table(’money’)
session.start_transaction()
money.update().set(’amt’, 4).where("name = ’Eve’").execute()
money.update().set(’amt’, 10).where("name = ’Ida’").execute()
session.commit()
74