The Hows and Whys of a Distributed SQL Database - Strange Loop 2017
•
4 likes•721 views
Until recently, developers have had to deal with some serious tradeoffs when picking a database technology. One could pick a SQL database and deal with their eventual scaling problems or pick a NoSQL database and have to work around their lack of transactions, strong consistency, and/or secondary indexes. However, a new class of distributed database engines is emerging that combines the transactional consistency guarantees of traditional relational databases with the horizontal scalability and high availability of popular NoSQL databases. In this talk, we'll examine the history of databases to see how we got here, covering the motivations for this new class of systems and why developers should care about them. We'll then take a deep dive into the key design choices behind one open source distributed SQL database, CockroachDB, that enable it to offer such properties and compare them to past SQL and NoSQL designs. We will look specifically at how to achieve the easy deployment and management of a scalable, self-healing, strongly-consistent database with techniques such as dynamic sharding and rebalancing, consensus protocols, lock-free transactions, and more.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The Hows and Whys of a Distributed SQL Database - Strange Loop 2017
Similar to The Hows and Whys of a Distributed SQL Database - Strange Loop 2017 (20)
Recently uploaded
Recently uploaded (20)
The Hows and Whys of a Distributed SQL Database - Strange Loop 2017
- 1. The Hows and Whys of a Distributed SQL Database Alex Robinson / Member of the Technical Staff alex@cockroachlabs.com / @alexwritescode
- 2. Agenda • Why a distributed SQL database? • How does a distributed SQL database work? ○ Data distribution ○ Data replication ○ Transactions
- 3. Why a distributed SQL database? A brief history of databases
- 4. 1960s: The first databases • Records stored with pointers to each other in a hierarchy or network • Queries had to process one tuple at a time, traversing the structure • No independence between logical and physical schemas ○ To optimize for different queries, you had to dump and reload all your data • Required a lot of programmer effort to use
- 5. 1970s: SQL / RDBMS • Relational model was designed in contrast to prior hierarchical DBs ○ Expose simple data structures and high-level language ○ Queries are independent of the underlying physical storage • Very developer friendly • Designed to run on a single machine
- 6. 1980s-1990s: Continued growth of SQL / RDBMS • SQL databases matured, took over the industry • Object-oriented databases tried, but didn’t gain much traction ○ Didn’t support complex queries well ○ No standard API across systems
- 7. Early 2000s: Custom Sharding • Rapidly growing scale necessitated new solutions • Companies added custom middleware to connect single-node DBs • No cross-shard transactions, operational nightmares MySQL Middleware Applications
- 8. 2004+: NoSQL • Further growth and operational pain necessitated new systems • Big change in priorities - scale and availability above all else • Sacrificed a lot to get there ○ No relational model (custom APIs instead of SQL) ○ No (or very limited) transactions and indexes ○ Manual joins
- 9. 2010s: “NewSQL” / Distributed SQL
- 10. Why another type of database?
- 11. Database Limitations Existing database solutions place an undue burden on application developers and operators: ● Scale (sql) ● Fault tolerance (sql) ● Limited transactions (nosql) ● Limited indexes (nosql) ● Consistency issues (nosql)
- 12. Strong Consistency and Transactions “We believe it is better to have application programmers deal with performance problems due to overuse of transactions as bottlenecks arise, rather than always coding around the lack of transactions” -Google’s Spanner paper
- 13. 2010s: “NewSQL” / Distributed SQL ● Distributed databases that provide full SQL semantics ● Attempt to combine the best of both worlds ○ Fully-featured SQL ○ Horizontally scalable and highly available ● Entirely new systems, tough to tack onto existing databases
- 14. 2010s: “NewSQL” / Distributed SQL ● Burst onto the scene with Google’s Spanner system (published in 2012) ● CockroachDB started open source development in 2014
- 15. So we know why, but how? And what’s different from SQL and NoSQL databases?
- 16. What we’re going to cover 1. How data is distributed across machines 2. How remote copies of data are replicated and kept in sync 3. How transactions work Plus how all of this differs from conventional SQL and NoSQL systems
- 17. Disclaimer
- 19. Data Distribution in SQL • Option 1: All data on one server ○ With optional secondary replicas/backups that also store all the data • Option 2: Manually shard data across separate database instances ○ All data for each shard is still just on one server (plus any secondary replicas) ○ All data distribution choices are being made by the human DB admin
- 20. Data Distribution in NoSQL / NewSQL • Core assumption: entire dataset won’t fit on one machine • Given that, there are two key questions to answer: ○ How do you divide it up? ○ How do you locate any particular piece of data? • Two primary approaches: Hashing or Order-Preserving
- 21. Option one: Hashing ● Pick one or more hash functions, divide up data by hashing each key ● Deterministically map hashed keys to servers ● Pros: easy to locate data by key ● Con: awkward, inefficient range scans
- 22. Option two: Order-Preserving ● Divide sorted key space up into ranges of approximately equal size ● Distribute the resulting key ranges across the servers ● Pros: efficient scans, easy splitting ● Con: requires additional indexing
- 23. Option two: Order-Preserving Each shard (“range”) contains a contiguous segment of the key space Ø-lem apricot banana blueberry cherry grape lem-pea lime mango melon orange pea-∞ peach pear pineapple raspberry strawberry lemon
- 24. Option two: Order-Preserving We need an indexing structure to locate a range Ø-lem apricot banana blueberry cherry grape lem-pea lime mango melon orange pea-∞ peach pear pineapple raspberry strawberry Ø-lem lem-pea pea-∞ lemon range index
- 25. Option two: Order-Preserving Ø-lem apricot banana blueberry cherry grape lem-pea lemon lime mango melon orange pea-∞ peach pear pineapple raspberry strawberry Ø-lem lem-pea pea-∞ range index Scans (fruits >= “cherry” AND <= “mango”) are efficient
- 26. Option two: Order-Preserving Ø-lem apricot banana blueberry cherry grape lem-pea lemon lime mango melon orange pea-str peach pear pineapple raspberry Ø-lem lem-pea range index str-∞ strawberry tamarillo tamarind str-∞pea-str
- 27. Data Distribution: Placement Node 1 Range 1 Range 2 Node 2 Range 1 Range 2 Node 3 Range 1 Range 2Range 3 Range 3 Range 2 Range 3 Each range is replicated to three or more nodes
- 28. Data Distribution: Rebalancing Node 1 Range 1 Range 2 Node 2 Range 1 Range 2 Node 3 Range 1 Node 4 Range 2Range 3 Range 3 Range 2 Range 3 Adding a new (empty) node
- 29. Data Distribution: Rebalancing Node 1 Range 1 Range 2 Node 2 Range 1 Range 2 Node 3 Range 1 Node 4 Range 2Range 3 Range 3 Range 2 Range 3 Range 3 A new replica is allocated, data is copied.
- 30. Data Distribution: Rebalancing Node 1 Range 1 Range 2 Node 2 Range 1 Range 2 Node 3 Range 1 Node 4 Range 2Range 3 Range 3 Range 2 Range 3 Range 3 The new replica is made live, replacing another.
- 31. Data Distribution: Rebalancing Node 1 Range 1 Range 2 Node 2 Range 1 Range 2 Node 3 Range 1 Node 4 Range 2Range 3 Range 3 Range 2 Range 3 The old (inactive) replica is deleted.
- 32. Data Distribution: Rebalancing Node 1 Range 1 Range 2 Range 2 Node 2 Node 3 Range 1 Node 4 Range 2Range 3 Range 1 Range 3 Range 2 Range 3 Process continues until nodes are balanced.
- 33. Data Distribution: Recovery Node 1 Range 1 Range 2 Range 2 Node 2 Node 3 Range 1 Node 4 Range 2Range 3 Range 1 Range 3 Range 2 Range 3 X Losing a node causes recovery of its replicas.
- 34. Data Distribution: Recovery Node 1 Range 1 Range 2 Range 2 Node 2 Node 3 Range 1 Node 4 Range 2Range 3 Range 1 Range 3 Range 2 Range 3 XRange 1 Range 3 A new replica gets created on an existing node to replace the lost replica.
- 35. Data Replication Keeping Remote Copies in Sync
- 36. Keeping Copies in Sync in SQL Databases • Option one: cold backups ○ No real expectation of backups being fully up-to-date • Option two: primary-secondary replication ○ All writes (and all reads that care about consistency) go to the “Primary” instance ○ All writes get pushed from the Primary to any Secondary instances
- 37. Primary/Secondary Replication Put “cherry” Put “cherry” Primary apricot banana blueberry cherry grape Secondary apricot banana blueberry cherry grape In theory, replicas contain identical copies of data: Voila!
- 38. Primary/Secondary Replication • In practice, you have to choose ○ Asynchronous replication ■ Secondaries lag behind primary ■ Failover is likely to lose recent writes ○ Synchronous replication ■ All writes get delayed waiting for secondary acknowledgment ■ What do you do when your secondary fails? • Failover is really hard to get right ○ How do clients know which instance is the primary at any given time?
- 39. Keeping Copies in Sync in NoSQL Databases • Most NoSQL systems are eventually consistent • Very wide space of solutions, many different possible behaviors ○ Last write wins ○ Last write wins with vector clocks ○ Leader elections ○ Quorum reads and writes ○ Conflict-free replicated data types (CRDTs)
- 40. Keeping Copies in Sync in NewSQL Databases • Use a distributed consensus protocol from the CS literature ○ Available protocols: Paxos, Raft, Zab, Viewstamped Replication, … • At a high level: ○ Replicate data to N (often N=3 or N=5) nodes ○ Commit happens when a quorum have written the data
- 41. Put “cherry” Leader apricot banana blueberry cherry grape Follower apricot banana blueberry grape Follower apricot banana blueberry grape Distributed Consensus: Raft made simple
- 42. Put “cherry” Leader apricot banana blueberry cherry grape Follower apricot banana blueberry grape Follower apricot banana blueberry grape Put “cherry” Distributed Consensus: Raft made simple
- 43. Write committed when 2 out of 3 nodes have written data Follower apricot banana blueberry cherry grape Follower apricot banana blueberry grape Distributed Consensus: Raft made simple Put “cherry” Leader apricot banana blueberry cherry grape Put “cherry”
- 44. Follower apricot banana blueberry cherry grape Follower apricot banana blueberry grape Distributed Consensus: Raft made simple Put “cherry” Leader apricot banana blueberry cherry grape Put “cherry” AckAck
- 45. Distributed Consensus: Raft made simple What happens during failover?
- 46. Distributed Consensus: Raft made simple Put “cherry” Leader apricot banana blueberry cherry grape Follower apricot banana blueberry grape Follower apricot banana blueberry grape
- 47. Distributed Consensus: Raft made simple Put “cherry” Leader apricot banana blueberry cherry grape Follower apricot banana blueberry grape Follower apricot banana blueberry grape
- 48. Distributed Consensus: Raft made simple Read “cherry” Leader apricot banana blueberry grape Follower apricot banana blueberry cherry grape Follower apricot banana blueberry grape
- 49. Distributed Consensus: Raft made simple On failover, only data written to a quorum is considered presentRead “cherry” Leader apricot banana blueberry grape Follower apricot banana blueberry cherry grape Follower apricot banana blueberry grape
- 50. Distributed Consensus: Raft made simple Read “cherry” Leader apricot banana blueberry grape Follower apricot banana blueberry cherry grape Follower apricot banana blueberry grape Key not found
- 51. Consensus in NewSQL Databases ● Run one consensus group per range of data ● Many practical complications: member change, range splits, upgrades, scaling number of ranges
- 52. Transactions How can they be made to work in a distributed system?
- 53. What is an ACID Transaction? • Atomic - entire transaction either fully commits or fully aborts • Consistent - moves from one valid DB state to another • Isolated - concurrent transactions don’t interfere with each other ○ “Serializable” is needed for true isolation • Durable - committed transactions won’t disappear
- 54. Transactions in SQL Databases • Atomicity is bootstrapped off a lower-level atomic primitive: log writes ○ All mutations applied as part of a transaction get tagged with a transaction ID ○ “Commit” log entry marks the transaction as committed, making mutations visible • Isolation is typically provided in one of two ways ○ Read/Write Locks ○ Multi-Version Concurrency Control (MVCC)
- 55. Transactions in SQL Databases: Read/Write Locks • Pretty simple at a high level: ○ When you read a row, take a read lock on it ○ When you write a row, take a write lock on it ○ When you commit, release all locks • Requires deadlock detection
- 56. Transactions in SQL Databases: MVCC • Store a timestamp with each row • When updating a row, don’t overwrite the old value • When reading, return the most recent value from before the read ○ Allows access to historical data ○ Allows long-running read-only transactions to not block new writes • Need to detect conflicts and abort conflicting transactions
- 57. Transactions in NoSQL Databases • Many NoSQL systems don’t offer transactions at all • This is one of the biggest tradeoffs that was made to improve scalability • Those that do typically limit transactions to a single key ○ Single-key transactions don’t require coordinating across shards ○ Can be implemented as just supporting a compare-and-swap operation
- 58. Transactions in NewSQL Databases • Support traditional ACID semantics • Atomicity is bootstrapped off distributed consensus ○ A “Commit” write to consensus system marks the transaction as committed • Isolation can be handled similarly to single-node SQL databases ○ But the added latency and distributed state makes things tougher
- 59. Distributed Transactions (CockroachDB) 1. Begin Txn 1 2. Put “cherry” apricot cherry (txn 1) grape Status: PENDING Txn-1 Record
- 60. Distributed Transactions (CockroachDB) 1. Begin Txn 1 2. Put “cherry” 3. Put “mango” apricot cherry (txn 1) grape Status: PENDING Txn-1 Record lemon mango (txn 1) orange
- 61. Distributed Transactions (CockroachDB) 1. Begin Txn 1 2. Put “cherry” 3. Put “mango” 4. Commit Txn 1 apricot cherry (txn 1) grape Status: COMMITTED Txn-1 Record lemon mango (txn 1) orange
- 62. Distributed Transactions (CockroachDB) 1. Begin Txn 1 2. Put “cherry” 3. Put “mango” 4. Commit Txn 1 5. Clean up intents apricot cherry grape Status: COMMITTED Txn-1 Record lemon mango orange
- 63. Distributed Transactions (CockroachDB) 1. Begin Txn 1 2. Put “cherry” 3. Put “mango” 4. Commit Txn 1 5. Clean up intents 6. Remove Txn 1 apricot cherry grape lemon mango orange
- 64. Distributed Transactions ● That’s the happy case ● What about conflicting transactions?
- 65. Distributed Transactions (write conflict) 1. Begin Txn 1 2. Put “cherry” apricot cherry (txn 1) grape Status: PENDING Txn-1 Record
- 66. Distributed Transactions (write conflict) 1. Begin Txn 1 2. Put “cherry” apricot cherry (txn 1) grape Status: PENDING Txn-1 Record 1. Begin Txn 2 2. Put “cherry” Status: PENDING Txn-2 Record
- 67. Distributed Transactions (write conflict) 1. Begin Txn 1 2. Put “cherry” apricot cherry (txn 1) grape Status: PENDING Txn-1 Record 1. Begin Txn 2 2. Put “cherry” a. Check Txn 1 record Status: PENDING Txn-2 Record
- 68. Distributed Transactions (write conflict) 1. Begin Txn 1 2. Put “cherry” apricot cherry (txn 1) grape Status: ABORTED Txn-1 Record 1. Begin Txn 2 2. Put “cherry” a. Check Txn 1 record b. Push Txn 1 Status: PENDING Txn-2 Record
- 69. Distributed Transactions (write conflict) 1. Begin Txn 1 2. Put “cherry” apricot cherry (txn 2) grape Status: ABORTED Txn-1 Record 1. Begin Txn 2 2. Put “cherry” a. Check Txn 1 record b. Push Txn 1 c. Update intent Status: PENDING Txn-2 Record
- 70. Distributed Transactions (write conflict) 1. Begin Txn 1 2. Put “cherry” apricot cherry (txn 2) grape Status: ABORTED Txn-1 Record 1. Begin Txn 2 2. Put “cherry” a. Check Txn 1 record b. Push Txn 1 c. Update intent 3. Commit Txn 2 Status: COMMITTED Txn-2 Record
- 71. Distributed Transactions 1 https://www.cockroachlabs.com/blog/serializable-lockless-distributed-isolation-cockroachdb/ 2 https://research.google.com/archive/spanner.html ● Transaction atomicity is bootstrapped on top of Raft atomicity ● This description omitted MVCC and other types of conflicts ○ More details on the Cockroach Labs blog1 or Spanner paper2
- 72. Summary
- 73. Summary ● Databases are best understood in the context in which they were built ● How does a NewSQL database work in comparison to previous DBs? ○ Distribute data to support large data sets and fault-tolerance ○ Replicate consistently with consensus protocols like Raft ○ Distributed transactions using consensus as the foundation