This talk provides an in-depth overview of the key concepts of Apache Calcite. It explores the Calcite catalog, parsing, validation, and optimization with various planners.

1. ><
INTRODUCTIONTO
INTRODUCTION TO APACHE CALCITE
APACHE CALCITE
JORDAN HALTERMAN
1

2. WHAT IS APACHE
CALCITE?
next 2

3. ><INTRODUCTION TO APACHE CALCITE 3
What is Apache Calcite?
• A framework for building SQL databases
• Developed over more than ten years
• Written in Java
• Previously known as Optiq
• Previously known as Farrago
• Became an Apache project in 2013
• Led by Julian Hyde at Hortonworks

4. ><INTRODUCTION TO APACHE CALCITE 4
Projects using Calcite
• Apache Hive
• Apache Drill
• Apache Flink
• Apache Phoenix
• Apache Samza
• Apache Storm
• Apache everything…

5. ><INTRODUCTION TO APACHE CALCITE 5
What is Apache Calcite?
• SQL parser
• SQL validation
• Query optimizer
• SQL generator
• Data federator

6. ><INTRODUCTION TO APACHE CALCITE
Parse
Queries are parsed using
a JavaCC generated
parser
Validate
Queries are validated
against known database
metadata
Optimize
Logical plans are optimized
and converted into physical
expressions
Execute
P h y s i c a l p l a n s a r e
converted into application-
specific executions
01 02 03 04
Stages of query execution
6

7. COMPONENTS next 7

8. ><INTRODUCTION TO APACHE CALCITE 8
Components of Calcite
• Catalog - Defines metadata and namespaces
that can be accessed in SQL queries
• SQL parser - Parses valid SQL queries into an
abstract syntax tree (AST)
• SQL validator - Validates abstract syntax trees
against metadata provided by the catalog
• Query optimizer - Converts AST into logical
plans, optimizes logical plans, and converts
logical expressions into physical plans
• SQL generator - Converts physical plans to
SQL

9. CATALOG next 9

10. ><INTRODUCTION TO APACHE CALCITE 10
Calcite Catalog
• Defines namespaces that can be accessed in Calcite
queries
• Schema
• A collection of schemas and tables
• Can be arbitrarily nested
• Table
• Represents a single data set
• Fields defined by a RelDataType
• RelDataType
• Represents fields in a data set
• Supports all SQL data types, including structs and

11. ><INTRODUCTION TO APACHE CALCITE 11
Schema
• A collection of schemas and tables
• Schemas can be arbitrarily nested

12. ><INTRODUCTION TO APACHE CALCITE 12
Schema
• A collection of schemas and tables
• Schemas can be arbitrarily nested

13. ><INTRODUCTION TO APACHE CALCITE 13
Table
• Represents a single data set
• Fields are defined by a RelDataType

14. ><INTRODUCTION TO APACHE CALCITE 14
Table
• Represents a single data set
• Fields are defined by a RelDataType

15. ><INTRODUCTION TO APACHE CALCITE 15
RelDataType
• Represents the data type of an object
• Supports all SQL data types, including
structs and arrays
• Similar to Spark’s DataType

16. ><INTRODUCTION TO APACHE CALCITE 16
RelDataType

17. ><INTRODUCTION TO APACHE CALCITE 17
RelDataType
data type enum

18. ><INTRODUCTION TO APACHE CALCITE 18
Statistic
• Provide table statistics used in optimization

19. ><INTRODUCTION TO APACHE CALCITE 19
Statistic
• Provide table statistics used in optimization

20. ><INTRODUCTION TO APACHE CALCITE 20
Usage of the Calcite catalog

21. ><INTRODUCTION TO APACHE CALCITE 21
Usage of the Calcite catalog
schema

22. ><INTRODUCTION TO APACHE CALCITE 22
Usage of the Calcite catalog
schema table

23. ><INTRODUCTION TO APACHE CALCITE 23
Usage of the Calcite catalog
schema table
data type

24. ><INTRODUCTION TO APACHE CALCITE 24
Usage of the Calcite catalog
schema table
data typedata type field

25. SQL PARSER next 25

26. ><INTRODUCTION TO APACHE CALCITE 26
Calcite SQL parser
• LL(k) parser written in JavaCC
• Input queries are parsed into an abstract
syntax tree (AST)
• Tokens are represented in Calcite by
SqlNode
• SqlNode can also be converted back to a
SQL string via the unparse method

27. ><INTRODUCTION TO APACHE CALCITE 27
JavaCC
• Java Compiler Compiler
• Created in 1996 at Sun Microsystems
• Generates Java code from a domain-
specific language
• ANTLR is the modern alternative used in
projects like Hive and Drill
• JavaCC has sparse documentation

28. ><INTRODUCTION TO APACHE CALCITE 28
JavaCC

29. ><INTRODUCTION TO APACHE CALCITE 29
JavaCC

30. ><INTRODUCTION TO APACHE CALCITE 30
JavaCC
tokens

31. ><INTRODUCTION TO APACHE CALCITE 31
JavaCC
tokens
or

32. ><INTRODUCTION TO APACHE CALCITE 32
JavaCC
tokens
or
function call

33. ><INTRODUCTION TO APACHE CALCITE 33
JavaCC
tokens
Java code
or
function call

34. ><INTRODUCTION TO APACHE CALCITE 34
SqlNode
• SqlNode represents an element in an
abstract syntax tree

35. ><INTRODUCTION TO APACHE CALCITE 35
SqlNode
• SqlNode represents an element in an
abstract syntax tree
select

36. ><INTRODUCTION TO APACHE CALCITE 36
SqlNode
• SqlNode represents an element in an
abstract syntax tree
identifiersselect

37. ><INTRODUCTION TO APACHE CALCITE 37
SqlNode
• SqlNode represents an element in an
abstract syntax tree
identifiersselect operator

38. ><INTRODUCTION TO APACHE CALCITE 38
SqlNode
• SqlNode represents an element in an
abstract syntax tree
identifiersselect operator identifier

39. ><INTRODUCTION TO APACHE CALCITE 39
SqlNode
• SqlNode represents an element in an
abstract syntax tree
identifiersselect operator identifier
data type

40. ><INTRODUCTION TO APACHE CALCITE 40
SqlNode
• SqlNode represents an element in an
abstract syntax tree
identifiersselect operator identifier
data type
identifier

41. ><INTRODUCTION TO APACHE CALCITE 41
SqlNode
• SqlNode’s unparse method converts a
SQL element back into a string

42. ><INTRODUCTION TO APACHE CALCITE 42
SqlNode
• SqlNode’s unparse method converts a
SQL element back into a string

43. ><INTRODUCTION TO APACHE CALCITE 43
SqlNode

44. ><INTRODUCTION TO APACHE CALCITE 44
SqlNode
• SqlDialect indicates the capitalization
and quoting rules of specific databases

45. ><INTRODUCTION TO APACHE CALCITE 45
SqlNode
• SqlDialect indicates the capitalization
and quoting rules of specific databases

46. QUERY OPTIMIZER next 46

47. ><INTRODUCTION TO APACHE CALCITE 47
Query Plans
• Query plans represent the steps necessary
to execute a query

48. ><INTRODUCTION TO APACHE CALCITE 48
Query Plans
• Query plans represent the steps necessary
to execute a query

49. ><INTRODUCTION TO APACHE CALCITE 49
Query Plans
• Query plans represent the steps necessary
to execute a query
table scan
table
scan

50. ><INTRODUCTION TO APACHE CALCITE 50
Query Plans
• Query plans represent the steps necessary
to execute a query
inner join table scan
table
scan

51. ><INTRODUCTION TO APACHE CALCITE 51
Query Plans
• Query plans represent the steps necessary
to execute a query
filter
inner join table scan
table
scan

52. ><INTRODUCTION TO APACHE CALCITE 52
Query Plans
• Query plans represent the steps necessary
to execute a query
filter
inner join
project
table scan
table
scan

53. ><INTRODUCTION TO APACHE CALCITE 53
Query Plans
• Query plans represent the steps necessary
to execute a query
filter
inner join
project
table scan
table
scan

54. ><INTRODUCTION TO APACHE CALCITE 54
Query Optimization
• Optimize logical plan
• Goal is typically to try to reduce the amount
of data that must be processed early in the
plan
• Convert logical plan into a physical plan
• Physical plan is engine specific and
represents the physical execution stages

55. ><INTRODUCTION TO APACHE CALCITE 55
Query Optimization
• Prune unused fields
• Merge projections
• Convert subqueries to joins
• Reorder joins
• Push down projections
• Push down filters

56. ><INTRODUCTION TO APACHE CALCITE 56
Query Optimization

57. ><INTRODUCTION TO APACHE CALCITE 57
Query Optimization

58. ><INTRODUCTION TO APACHE CALCITE 58
Query Optimization

59. ><INTRODUCTION TO APACHE CALCITE 59
Query Optimization
push down
project

60. ><INTRODUCTION TO APACHE CALCITE 60
Query Optimization
push down
project
push down
filter

61. ><INTRODUCTION TO APACHE CALCITE 61
Query Optimization

62. ><INTRODUCTION TO APACHE CALCITE 62
Key Concepts
Relational algebra
Row expressions
Traits
Conventions
Rules
Planners
Programs

63. ><INTRODUCTION TO APACHE CALCITE 63
Key Concepts
Relational algebra
Row expressions
Traits
Conventions
Rules
Planners
Programs
RelNode
RexNode
RelTrait
Convention
RelOptRule
RelOptPlanner
Program

64. ><INTRODUCTION TO APACHE CALCITE 64
Relational Algebra
• RelNode represents a relational expression
• Largely equivalent to Spark’s DataFrame
methods
• Logical algebra
• Physical algebra

65. ><INTRODUCTION TO APACHE CALCITE 65
Relational Algebra
TableScan
Project
Filter
Aggregate
Join
Union
Intersect
Sort

66. ><INTRODUCTION TO APACHE CALCITE 66
Relational Algebra
TableScan
Project
Filter
Aggregate
Join
Union
Intersect
Sort
SparkTableScan
SparkProject
SparkFilter
SparkAggregate
SparkJoin
SparkUnion
SparkIntersect
SparkSort

67. ><INTRODUCTION TO APACHE CALCITE 67
Row Expressions
• RexNode represents a row-level expression
• Largely equivalent to Spark’s Column
functions
• Projection fields
• Filter condition
• Join condition
• Sort fields

68. ><INTRODUCTION TO APACHE CALCITE 68
Row Expressions
Input column ref
Literal
Struct field access
Function call
Window expression

69. ><INTRODUCTION TO APACHE CALCITE 69
Row Expressions
Input column ref
Literal
Struct field access
Function call
Window expression
RexInputRef
RexLiteral
RexFieldAccess
RexCall
RexOver

70. ><INTRODUCTION TO APACHE CALCITE 70
Row Expressions

71. ><INTRODUCTION TO APACHE CALCITE 71
Row Expressions
input ref

72. ><INTRODUCTION TO APACHE CALCITE 72
Row Expressions
input ref
function call

73. ><INTRODUCTION TO APACHE CALCITE 73
Traits
• Defined by the RelTrait interface
• Represent a trait of a relational expression
that does not alter execution
• Traits are used to validate plan output
• Three primary trait types:
• Convention
• RelCollation
• RelDistribution

74. ><INTRODUCTION TO APACHE CALCITE 74
Conventions
• Convention is a type of RelTrait
• A Convention is associated with a
RelNode interface
• SparkConvention, JdbcConvention,
EnumerableConvention, etc
• Conventions are used to represent a single
data source
• Inputs to a relational expression must be in
the same convention

75. ><INTRODUCTION TO APACHE CALCITE 75
Conventions

76. ><INTRODUCTION TO APACHE CALCITE 76
Conventions
Spark convention

77. ><INTRODUCTION TO APACHE CALCITE 77
Conventions
Spark convention
JDBC
convention

78. ><INTRODUCTION TO APACHE CALCITE 78
Conventions
Spark convention
JDBC
convention
converter

79. ><INTRODUCTION TO APACHE CALCITE 79
Rules
• Rules are used to modify query plans
• Defined by the RelOptRule interface
• Two types of rules: converters and
transformers
• Converter rules implement Converter and
convert from one convention to another
• Rules are matched to elements of a query
plan using pattern matching
• onMatch is called for matched rules
• Converter rules applied via convert

80. ><INTRODUCTION TO APACHE CALCITE 80
Converter Rule

81. ><INTRODUCTION TO APACHE CALCITE 81
Converter Rule
expression type

82. ><INTRODUCTION TO APACHE CALCITE 82
Converter Rule
expression type
input convention

83. ><INTRODUCTION TO APACHE CALCITE 83
Converter Rule
expression type
input convention
converted convention

84. ><INTRODUCTION TO APACHE CALCITE 84
Converter Rule
expression type
input convention
converted convention
converter function

85. ><INTRODUCTION TO APACHE CALCITE 85
Pattern Matching

86. ><INTRODUCTION TO APACHE CALCITE 86
Pattern Matching

87. ><INTRODUCTION TO APACHE CALCITE 87
Pattern Matching
no match
:-(

88. ><INTRODUCTION TO APACHE CALCITE 88
Pattern Matching
no match
:-(

89. ><INTRODUCTION TO APACHE CALCITE 89
Pattern Matching
match!
no match
:-(

90. ><INTRODUCTION TO APACHE CALCITE 90
Planners
• Planners implement the RelOptPlanner
interface
• Two types of planners:
• HepPlanner
• VolcanoPlanner

91. ><INTRODUCTION TO APACHE CALCITE 91
Heuristic Optimization
• HepPlanner is a heuristic optimizer similar
to Spark’s optimizer
• Applies all matching rules until none can be
applied
• Heuristic optimization is faster than cost-
based optimization
• Risk of infinite recursion if rules make
opposing changes to the plan

92. ><INTRODUCTION TO APACHE CALCITE 92
Cost-based Optimization
• VolcanoPlanner is a cost-based
optimizer
• Applies matching rules iteratively, selecting
the plan with the cheapest cost on each
iteration
• Costs are provided by relational expressions
• Not all possible plans can be computed
• Stops optimization when the cost does not
significantly improve through a determinable
number of iterations

93. ><INTRODUCTION TO APACHE CALCITE 93
Cost-based Optimization
• Cost is provided by each RelNode
• Cost is represented by RelOptCost
• Cost typically includes row count, I/O, and
CPU cost
• Cost estimates are relative
• Statistics are used to improve accuracy of
cost estimations
• Calcite provides utilities for computing
various resource-related statistics for use in
cost estimations

94. ><INTRODUCTION TO APACHE CALCITE 94
Cost-based Optimization

95. ><INTRODUCTION TO APACHE CALCITE 95
Cost-based Optimization

96. ><INTRODUCTION TO APACHE CALCITE 96
Cost-based Optimization

97. ><INTRODUCTION TO APACHE CALCITE 97
Cost-based Optimization

98. ><INTRODUCTION TO APACHE CALCITE 98
Cost-based Optimization

99. PUTTING IT ALL
TOGETHER
next 99

100. ><INTRODUCTION TO APACHE CALCITE 100
Putting it all together

101. ><INTRODUCTION TO APACHE CALCITE 101
Putting it all together

102. ><INTRODUCTION TO APACHE CALCITE 102
Putting it all together

103. ><INTRODUCTION TO APACHE CALCITE 103
Putting it all together

104. ><INTRODUCTION TO APACHE CALCITE 104
Putting it all together

105. ><INTRODUCTION TO APACHE CALCITE 105
Putting it all together

106. ><INTRODUCTION TO APACHE CALCITE 106
Putting it all together

107. ><INTRODUCTION TO APACHE CALCITE 107
Putting it all together

108. ><INTRODUCTION TO APACHE CALCITE 108
Putting it all together

109. ><INTRODUCTION TO APACHE CALCITE 109
Putting it all together