Presto query optimizer: pursuit of performance
The talk from DataWorks Summit 2018 in San Jose, CA
https://dataworkssummit.com/san-jose-2018/session/presto-query-optimizer-pursuit-of-performance/
Folow us at @starburstdata and our blog:
https://www.starburstdata.com/technical-blog/
6. Presto in Production
Facebook: 1000s of nodes, HDFS (ORC, RCFile), sharded MySQL, 1000s of users
Uber: 800+ nodes (2 clusters on premises) with 200K+ queries daily over HDFS (Parquet/ORC)
Twitter: 800+ nodes (several clusters on premises) for HDFS (Parquet)
LinkedIn: 350+ nodes (2 clusters on premises), 40K+ queries daily over HDFS (ORC), 600+ users
Netflix: 250+ nodes in AWS, 40+ PB in S3 (Parquet)
Lyft: 200+ nodes in AWS, 20K+ queries daily, 20+ PBs in Parquet
Yahoo! Japan: 200+ nodes (4 clusters on premises) for HDFS (ORC), ObjectStore, and Cassandra
FINRA: 120+ nodes in AWS, 4PB in S3 (ORC), 200+ users
7. Built for Performance
Query Execution Engine:
● MPP-style pipelined in-memory execution
● Columnar and vectorized data processing
● Runtime query bytecode compilation
● Memory efficient data structures
● Multi-threaded multi-core execution
● Optimized readers for columnar formats (ORC and Parquet)
● Now also Cost-Based Optimizer
8. Evolving the optimizer - Challenges
● Diverse and widespread production workloads
● Fast-changing codebase
● Many developers
● Large surface area and usage of plan IR
9. Before
● Monolithic visitor-based plan transformations
● Visitors responsible for walking and transforming plan tree
● Problems
○ Hard to add new operations (IR node types)
○ Hard to add new optimizations
○ Hard to test optimizers
class LimitPushdown {
Plan optimize(Plan) { return plan.root.accept(this) }
Node visitLimit(LimitNode) { ... }
Node visitProject(ProjectNode) { ... }
Node visitFilter(FilterNode) { ... }
...
}
10.
11. Now
● Granular rule-based transformations
● Rules responsible for transforming localized subplan structure
● Optimizer loop responsible for walking plan and driving rule application
● Benefits
○ Decouples traversal from rule application
○ Decouples adding new optimizations from adding new operations (IR node types)
○ Easier to reason about and test individual rule behavior
class PushLimitThroughProjectRule {
Pattern getPattern() { Patterns.limit().with(source().matching(project())) }
Node apply(Node) { ... }
}
12.
13.
14.
15.
16. Migrating from monolithic optimizers to rules
● Fallback behavior
● Controlled via config option or
per-query session property
● Removed after a few releases
optimizers = [
RuleBasedOptimizer(
legacy = LimitPushdown,
rules = [
PushLimitThroughProject,
PushLimitThroughUnion,
PushLimitThroughJoin
]
),
PredicatePushdown,
PruneUnusedColumns,
AddExchanges,
EliminateCrossJoins,
...
]
17. Adding cost-aware optimizers
● Just another rule
● Can reason about cost
optimizers = [
RuleBasedOptimizer(
rules = [
PushLimitThroughProject,
PushLimitThroughUnion,
PushLimitThroughJoin,
ReorderJoins
]
),
...
]
class ReorderJoins {
ReorderJoins(CostComparator) { ... }
Pattern getPattern() { ... }
Node apply(Node) { ... }
}
18. CBO in a nutshell
Cost-Based Optimizer v1 includes:
● support for statistics stored in Hive Metastore
● join reordering based on selectivity estimates and cost
● automatic join type selection (repartitioned vs broadcast)
● automatic left/right side selection for joined tables
https://www.starburstdata.com/technical-blog/
19. Statistics & Cost
Hive Metastore statistics:
● number of rows in a table
● number of distinct values in a column
● fraction of NULL values in a column
● minimum/maximum value in a column
● average data size for a column
Cost calculation includes:
● CPU
● Memory
● Network I/O
