The Power of the Log: LSM & Append Only Data Structures
•
20 likes•6,841 views
LSM trees provide an efficient way to structure databases by organizing data sequentially in logs. They optimize for write performance by batching writes together sequentially on disk. To optimize reads, data is organized into levels and bloom filters and caching are used to avoid searching every file. This log-structured approach works well for many systems by aligning with how hardware is optimized for sequential access. The immutability of appended data also simplifies concurrency. This log-centric approach can be applied beyond databases to distributed systems as well.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to The Power of the Log: LSM & Append Only Data Structures
Similar to The Power of the Log: LSM & Append Only Data Structures (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
The Power of the Log: LSM & Append Only Data Structures
- 1. The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc
- 2. @benstopford
- 3. The Log ConnectorsConnectors Producer Consumer Streaming Engine Kafka: a Streaming Platform
- 4. KAFKA’s Distributed Log Linear ScansAppend Only
- 5. Messaging is a Log-Shaped Problem Linear ScansAppend Only
- 6. Not all problems are Log-Shaped
- 7. Many problems benefit from being addressed in a “log-shaped” way
- 9. Lookups in a log HeadTail
- 10. Trees provide Selectivity bob dave fred hary mike steve vince Index
- 11. But the overarching structure implies Dispersed Writes bob dave fred hary mike steve vince Random IO
- 12. Log Structured Merge Trees 1996
- 13. Used in a range of modern databases • BigTable • HBase • LevelDB • SQLite4 • RocksDB • MongoDB • WiredTiger • Cassandra • MySQL • InfluxDB ...
- 14. If a systems have a natural grain, it is one formed of sequential operations which favour locality
- 15. Caching & Prefetching L3 cache L2 cache L1 cache Pre-fetch is your friend CPU Caches Page Cache Application-level caching Disk Controller
- 16. Write efficiency comes from amortising writes into sequential operations
- 17. Taken from ACMQueue: The Pathologies of Big Data
- 18. So if we go against the grain of the system, RAM can actually be slower than disk
- 19. Going against the grain means dispersed operations that break locality Poor Locality Good Locality
- 20. The beauty of the log lies in its sequentially Linear ScansAppend Only
- 21. LSM is about re-imagining search as as a “log-shaped” problem
- 22. Arrange writes to be Append Only Append Only Journal (Sequential IO) Update in Place Ordered File (Random IO) Bob = Carpenter Bob = Carpenter Bob = Cabinet Maker Bob = Cabinet Maker
- 24. Simple LSM
- 25. Writes are collected in memory Writes sort write to disk older files small index file RAM
- 26. When enough have buffered, sort. Writes write to disk older files small index file Batched sorted RAM
- 27. Write the sorted file to disk Writes write to disk older files Small, sorted immutable file Batched sorted
- 28. Repeat... Writes write to disk Older files New files Batched sorted
- 29. Batching -> Fast Sequential IO Writes write to disk Older files New files Batched Sorted memtable
- 30. That’s the core write path
- 32. Search reverse-chronologically older files newer files (1) Is “bob” here? (2) Is “bob” here? (3) Is “bob” here? (4) Is “bob” here?
- 33. Worst Case We consult every file
- 34. We might have a lot of files!
- 35. LSM naturally optimises for writes, over reads This is a reasonable tradeoff to make
- 36. Optimizing reads is easier than optimising writes
- 37. Optimisation 1 Bound the number of files
- 39. Separate thread merges old files, de- duplicating them. Level-0 Level-1
- 40. Separate thread merges old files, de- duplicating them. Level-0 Level-1
- 41. Merging process is reminiscent of merge sort
- 42. Take this further with levels Level-0 Level-1 Level-2 Level-3 Memtable
- 43. But single reads still require many individual lookups: • Number of searches: – 1 per base level – 1 per level above
- 44. Optimisation 2 Caching & Friends
- 45. Add Memory i.e. More Caching / Pre-fetch
- 46. Read Ahead & Prefetch L3 cache L2 cache L1 cache Pre-fetch is your friend Page Cache Disk Controller
- 47. If only there was a more efficient way to avoid searching each file!
- 48. Elven Magic?
- 49. Bloom Filters Answers the question: Do I need to look in this file to find the value for this key? Size -> probability of false positive Key Hash Function Bit Set
- 50. Bloom Filters • Space efficient, probabilistic data structure • As keyspace grows: – p(collision) increases – Index size is fixed
- 51. Many more degrees of freedom for optimising reads RAM Disk file metadata & bloom filter
- 52. Log Structured Merge Trees • A collection of small, immutable indexes • All sequential operations, de-duplicate by merging files • Index/Bloom in RAM to increase read performance
- 53. Subtleties • Writes are 1 x IO (blind writes) , rather than 2 x IO’s (read + modify) • Batching writes decreases write amplification. In trees leaf pages must be updated.
- 54. Immutability => Simpler locking semantics Only memtable is mutable
- 55. Does it work? Lots of real world examples
- 56. Measureable in the real world • Innodb vs MyRocks results, taken from Mark Callaghan’s blog: http://bit.ly/2mhWT7p • There are many subtleties. Take all benchmarks with a pinch of salt.
- 57. Elements of Beauty • Reframing the problem to be Log-Centric. To go with the grain of the system. • Optimise for the harder problem • Compartmentalises writes (coordination) to a single point. Reads -> immutable structures.
- 58. Applies in many other areas • Sequentiality – Databases: write ahead logs – Columnar databases: Merge Joins – Kafka • Immutability – Snapshot isolation over explicit locking. – Replication (state machines replication)
- 59. Log-Centric Approaches Work in Applications too
- 60. Event Sourcing • Journaling of state changes • No “update in place” Object Journal + 10.36 - 12.12 + 23.70 + 13.33
- 62. How Applications or Services share state
- 63. Log-Centric Services Writer Read-Replica Read-Replica Read-Replica Writes are localised to a single service
- 65. Log-Centric Services Writer Read-Replica Read-Replica Read-Replica Many, independent read replicas
- 66. Elements of Beauty • Reframing the problem to be Log-Centric. To go with the grain of the system. • Optimise for the harder problem • Compartmentalises writes (coordination) to a single point. Reads -> immutable structures.
- 67. Decentralised Design In both database design as well as in application development
- 68. The Log is the central building block Pushes us towards the natural grain of the system
- 69. The Log A single unifying abstraction
- 70. References LSM: • benstopford.com/2015/02/14/log-structured-merge-trees/ • smalldatum.blogspot.co.uk/2017/02/using-modern-sysbench-to-compare.html • www.quora.com/How-does-the-Log-Structured-Merge-Tree-work • bLSM paper: http://bit.ly/2mT7Vje Other • Pat Helland (Immutability) cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf • Peter Ballis (Coordination Avoidance): http://bit.ly/2m7XxnI • Jay Kreps: I Heart Logs (O’Reilly 2014) • The Data Dichotomy: http://bit.ly/2hk9c2K