An Introduction to TiDB for the San Francisco MySQL Meetup Group

1. Introducing TiDB
(For those coming from MySQL..)
Make Data Creative
Morgan Tocker (@pingcap; @morgo)
October, 2018

2. ● History and Community
● Technical Walkthrough
● Use Case with Mobike
● Live Demo: TiDB on GKE
● MySQL Compatibility
● Q&A
Agenda

3. ● Sr Product / Community Manager
● ~15+ years MySQL Experience
○ MySQL AB, Sun Microsystems, Percona, Oracle
● Previously Product Manager for MySQL Server
A Little About Me...

4. A Little About PingCAP...
● Founded in April 2015 by 3 infrastructure engineers
● Offices in China and North America
● Remote Friendly!
○ I work from here ➡

5. Recent News

6. Recent News

7. Our Product is The TiDB Platform
● TiDB platform: (Ti = Titanium)
○ TiDB (stateless SQL layer compatible with MySQL)
○ TiKV (distributed transactional key-value store)
○ TiSpark (Apache Spark plug-in on top of TiKV)
● Open source from Day 1
○ GA 1.0: October 2017
○ GA 2.0: April 2018

8. TiDB is a NewSQL Database
● 1960s: First Gen Databases
● 1970s: Relational Model + SQL
● 2000s: Sharding + Memcached
● 2010s: NoSQL
● 2010s+: NewSQL
Inspired by Google Spanner / F1
MySQL Compatible

9. ● Hybrid OLTP & OLAP (Minimize ETL)
● Horizontal Scalability
● MySQL Compatible
● Distributed Transaction (ACID Compliant)
● High Availability
● Cloud-Native
TiDB Core Features

10. Architecture
SparkSQL
TiDB
TiDB
Worker
Spark
Driver
TiKV Cluster (Storage)
Metadata
TiKV TiKV
TiKV
Data location
Job
TiSpark
DistSQL API
TiKV
TiDB
TSO/Data location
Worker
Worker
Spark Cluster
TiDB Cluster
TiDB
DistSQL API
PD
PD Cluster
TiKV TiKV
TiDB
KV API
MySQL
MySQL
PD
PD

11. 2018 PingCAP
Stars
● TiDB: 15,000+
● TiKV: 3700+
Contributors
● TiDB: 200+
● TiKV: 100+
Community

12. Early Sign-up: https://www.pingcap.com/tidb-academy/
Sneak Peek!

13. TiDB Platform Architecture

14. Platform Architecture
TiDB
TiDB
Worker
Spark
Driver
TiKV Cluster (Storage)
Metadata
TiKV TiKV
TiKV
Data location
Job
TiSpark
DistSQL API
TiKV
TiDB
TSO/Data location
Worker
Worker
Spark Cluster
TiDB Cluster
TiDB
DistSQL API
PD
PD Cluster
TiKV TiKV
TiDB
KV API
MySQL
MySQL
SparkSQL
PD
PD
SparkSQL

15. TiKV: The Foundation [in CNCF]
RocksDB
Raft
Transaction
Txn KV API
Coprocessor
API
RocksDB
Raft
Transaction
Txn KV API
Coprocessor
API
RocksDB
Raft
Transaction
Txn KV API
Coprocessor
API
Raft
Group
Client
gRPC
TiKV Instance TiKV Instance TiKV Instance
gRPC gRPC
PD Cluster

16. TiDB: OLTP + Ad Hoc OLAP
Node1 Node2 Node3 Node4
MySQL Network Protocol
SQL Parser
Cost-based Optimizer
Distributed Executor (Coprocessor)
ODBC/JDBC MySQL Client
Any ORM which
supports MySQL
TiDB
TiKV

17. ID Name Email
1 Edward h@pingcap.com
2 Tom tom@pingcap.com
...
user/1 Edward,h@pingcap.com
user/2 Tom,tom@pingcap.com
...
In TiKV -∞
+∞
(-∞, +∞)
Sorted map
“User” Table
TiDB: Relational -> KV
Some region...

18. SQL -> Parser -> Coprocessor

19. TiSpark: Complex OLAP
Spark ExecSpark Exec
Spark Driver
Spark Exec
TiKV TiKV TiKV TiKV
TiSpark
TiSpark TiSpark TiSpark
TiKV
Placement
Driver (PD)
gRPC
Distributed Storage Layer
gRPC
retrieve data location
retrieve real data from TiKV

20. Who’s Using TiDB?

21. 2018 PingCAP
Who’s using TiDB?
300+
Companies

22. 2018 PingCAP
1. MySQL Scalability
2. Hybrid OLTP/OLAP Architecture
3. Unifying Data Storage/Management
Three Big Use Cases

23. Mobike + TiDB
● 200 million users
● 200 cities
● 9 milllion smart bikes
● ~30 TB / day

24. ● Locking and unlocking of smart bikes generate massive data
● Smooth experience is key to user retention
● TiDB supports this system by alerting administrators when
success rate of locking/unlocking drops, within minutes
● Quickly find malfunctioning bikes
Scenario #1: Locking/Unlocking

25. ● Synchronize TiDB with MySQL
instances using Syncer (proprietary
tool)
● TiDB + TiSpark empower real-time
analysis with horizontal scalability
● No need for Hadoop + Hive
Scenario #2: Real-Time Analysis

26. ● An innovative loyalty program that must
be on 24 x 7 x 365
● TiDB handles:
○ High-concurrency for peak or promotional season
○ Permanent storage
○ Horizontal scalability
● No interruption as business evolves
Scenario #3: Mobike Store

27. TiDB on GKE Demo

28. MySQL Compatibility

29. ● Compatible with MySQL 5.7
○ Joins, Subqueries, DML, DDL etc.
● On the roadmap:
○ Views, Window Functions, GIS
● Missing:
○ Stored Procedures, Triggers, Events
Summary
pingcap.com
/docs/sql/mysql-compatibility/

30. ● Some features work differently
○ Auto Increment
○ Optimistic Locking
● TiDB works better with smaller
transactions
○ Recommended to batch updates, deletes,
inserts to 5000 rows
Nuanced

31. Thank You!
Twitter: @PingCAP; @morgo
https://github.com/pingcap
(Give us a Watch/Star!)
Morgan Tocker
(morgan@pingcap.com)
Early Sign-up:
www.pingcap.com/tidb-academy/

32. ● Hash Join (fastest; if table <= 50 million rows)
● Sort Merge Join (join on indexed column or ordered data
source)
● Index Lookup Join (join on indexed column; ideally after filter,
result < 10,000 rows)
Chosen based on Cost-based Optimizer:
Join Support
Network cost Memory cost CPU cost

33. Index Structure
Row:
Key: tablePrefix_rowPrefix_tableID_rowID (IDs are assigned by TiDB, all int64)
Value: [col1, col2, col3, col4]
Index:
Key: tablePrefix_idxPrefix_tableID_indexID_ColumnsValue_rowID
Value: [null]
Keys are ordered by byte array in TiKV, so can support SCAN
Every key is appended a timestamp, issued by Placement Driver

34. ● Complex calculation pushdown
● Key-range pruning
● Index support:
○ Clustered index / non-clustered index
○ Index-only query optimization
● Cost-based optimization:
○ Stats gathered from TiDB in histogram
TiSpark: Features

35. PD: Dynamic Split and Merge
Region A
Region A
Region B
Region A
Region A
Region B
Split
Region A
Region A
Region B
Merge
TiKV_1 TiKV_2 TiKV_2TiKV_1

36. PD: Hotspot Removal
*Region A*
*Region B*
Region A
Region B
Workload
*Region A*
Region B
Region A
*Region B*
Workload
Workload
Hotspot Schedule
(Raft leader transfer)
TiKV_1 TiKV_2
TiKV_2TiKV_1

37. Geo-Replication + Data Location
*Region A*
Region B
Region A
Region B
Seattle_1 Seattle_2
Region A
*Region B*
New York_1
*Region A*
Region B
Region A
*Region B*
Seattle_2Seattle_1
Region A
Region B
New York_1

38. ● Timestamp Oracle service (from Google’s Percolator paper)
● 2-Phase commit protocol (2PC)
● Problem: Single point of failure
● Solution: Placement Driver HA cluster
○ Replicated using Raft
Transaction Model

39. ● Formal proof using TLA+
○ a formal specification and verification language to reason about and prove
aspects of complex systems
● Raft
● TSO/Percolator
● 2PC
● See details: https://github.com/pingcap/tla-plus
Guaranteeing Correctness