Large partitions shall no longer be a nightmare. That is the goal of CASSANDRA-11206.
100MB and 100,000 cells per partition is the recommended limit for a single partition in Cassandra up to 3.5. Exceeding these limits can cause a lot of trouble. Repairs and compactions could fail and reads cause out-of-memory failures.
This talk provides a deep-dive of the reasons for the previous limitations, why exceeding these limitations caused trouble, how the improvements in Cassandra 3.6 helps with big partitions and why you should not blindly let your partitions get huge.
About the Speaker
Robert Stupp Solution Architect, DataStax
Robert is working as a Solutions Architect at DataStax and is also a Committer to Apache Cassandra. Before joining DataStax he worked with his customers to architect and build distributed systems using Cassandra and has a long experience in building distributed backend systems mostly using Java as the preferred language of choice.
POLL:
C* 2.x
C* 3.0.x
C* < 3.6
C* >= 3.6
Agenda
Why big partitions hurt before 3.6
What CASSANDRA-11206 changed
Things to consider
Big means many MB - couple 100s and more - like GB
Going to explain why this is an issue w/ C* < 3.6
Compression-Info – Digest – Statistics – TOC
Compression-Info – Digest – Statistics – TOC
Compression-Info – Digest – Statistics – TOC
How a primary index file looks like
Read "comes" from Summary
Scan until requested Partition is found
Emphasize Java Objects
Issues
Many objects
Promoted between Java heap generations
Evicted key cache entries usually in old gen
Huge GC pressure
Affects
Reads
Compaction
Repair
Flush (usually not an issue)
IndexedEntry including all IndexInforead as Java object structure
IndexedEntry including all IndexInfoplaced in key cache
DONE for ALL kinds of reads
Global
- all sstables
- all tables
- all keyspaces
Partition sizes: 16k, 1M, 4M, 8M … 512M, 1G, 2G, 4G, 8G with varying total data sizes