A Deep Dive into Query Execution Engine of Spark SQL
•
22 likes•6,735 views
Spark SQL enables Spark to perform efficient and fault-tolerant relational query processing with analytics database technologies. The relational queries are compiled to the executable physical plans consisting of transformations and actions on RDDs with the generated Java code. The code is compiled to Java bytecode, executed at runtime by JVM and optimized by JIT to native machine code at runtime. This talk will take a deep dive into Spark SQL execution engine. The talk includes pipelined execution, whole-stage code generation, UDF execution, memory management, vectorized readers, lineage based RDD transformation and action.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to A Deep Dive into Query Execution Engine of Spark SQL
Similar to A Deep Dive into Query Execution Engine of Spark SQL (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
A Deep Dive into Query Execution Engine of Spark SQL
- 1. Deep Dive: Query Execution of Spark SQL Maryann Xue, Xingbo Jiang, Kris Mok Apr. 2019 1
- 2. About Us Software Engineers • Maryann Xue PMC of Apache Calcite & Apache Phoenix @maryannxue • Xingbo Jiang Apache Spark Committer @jiangxb1987 • Kris Mok OpenJDK Committer @rednaxelafx 2
- 3. DATABRICKS WORKSPACE Databricks Delta ML Frameworks DATABRICKS CLOUD SERVICE DATABRICKS RUNTIME Reliable & Scalable Simple & Integrated Databricks Unified Analytics Platform APIs Jobs Models Notebooks Dashboards End to end ML lifecycle
- 4. Databricks Customers Across Industries Financial Services Healthcare & Pharma Media & Entertainment Technology Public Sector Retail & CPG Consumer Services Energy & Industrial IoTMarketing & AdTech Data & Analytics Services
- 5. Apache Spark 3.x 5 Catalyst Optimization & Tungsten Execution SparkSession / DataFrame / Dataset APIs SQL Spark ML Spark Streaming Spark Graph 3rd-party Libraries Spark CoreData Source Connectors
- 6. Apache Spark 3.x 6 Catalyst Optimization & Tungsten Execution SparkSession / DataFrame / Dataset APIs SQL Spark ML Spark Streaming Spark Graph 3rd-party Libraries Spark CoreData Source Connectors
- 7. Spark SQL Engine 7 Analysis -> Logical Optimization -> Physical Planning -> Code Generation -> Execution Runtime
- 8. Spark SQL Engine - Front End 8 Analysis -> Logical Optimization -> Physical Planning -> Code Generation -> Execution Reference: A Deep Dive into Spark SQL’s Catalyst Optimizer, Yin Huai, Spark Summit 2017 Runtime
- 9. Spark SQL Engine - Back End 9 Analysis -> Logical Optimization -> Physical Planning -> Code Generation -> Execution Runtime
- 10. Agenda 10
- 12. • Transform logical operators into physical operators • Choose between different physical alternatives - e.g., broadcast-hash-join vs. sort-merge-join • Includes physical traits of the execution engine - e.g., partitioning & ordering. • Some ops may be mapped into multiple physical nodes - e.g., partial agg —> shuffle —> final agg Physical Planning 1
- 13. A Physical Plan Example 1 Scan A Scan B Filter BroadcastExchange BroadcastHashJoin HashAggregate ShuffleExchange HashAggregate SELECT a1, sum(b1)FROM A JOIN B ON A.key = B.key WHERE b1 < 1000 GROUP BY a1 Scan A Filter Join Aggregate Scan B
- 14. • Scalar subquery Broadcast exchange: - Executed as separate jobs • Partition-local ops: - Executed in the same stage • Shuffle: - The stage boundary - A sync barrier across all nodes Scheduling a Physical Plan 1 Job 2 Stage 1 Stage 2 Scan A BroadcastHashJoin HashAggregate ShuffleExchange HashAggregate Job 1 Stage 1 Scan B Filter BroadcastExchange
- 16. Execution, Old: Volcano Iterator Model • Volcano iterator model - All ops implement the same interface, e.g., next() - next() on final op -> pull input from child op by calling child.next() -> goes on and on, ending up with a propagation of next() calls • Pros: Good abstraction; Easy to implement • Cons: Virtual function calls —> less efficient 1 Scan Filter Project Result Iterator iterate next()next()next()
- 17. Execution, New: Whole-Stage Code Generation • Inspired by Thomas Neumann’s paper • Fuse a string of operators (oftentimes the entire stage) into one WSCG op that runs the generated code. • A general-purpose execution engine just like Volcano model but without Volcano’s performance downsides: - No virtual function calls - Data in CPU registers - Loop unrolling & SIMD Scan Filter Project Aggregate long count = 0; for (item in sales) { if (price < 100) { count += 1; } }
- 18. Execution Models: Old vs. New • Volcano iterator model: Pull model; Driven by the final operator 1 Scan Filter Project Result Iterator iterate next()next()next() • WSCG model: Push model; Driven by the head/source operator Scan Filter Project Result Iterator iterate next()
- 19. A Physical Plan Example - WSCG Job 2 Stage 1 Stage 2 WSCG Scan A BroadcastHashJoin WSCG HashAggregate ShuffleExchange HashAggregate Job 1 Stage 1 WSCG Scan B Filter BroadcastExchange
- 20. Implementation • The top node WholeStageCodegenExec implements the iterator interface to interop with other code-gen or non-code-gen physical ops. • All underlying operators implement a code-generation interface: doProduce() & doConsume() • Dump the generated code: df.queryExecution.debug.codegen
- 21. WSCG Pipeline 2Pipeline 1 Single dependency • A WSCG node contains a linear list of physical operators that support code generation. • No multi dependency between enclosed ops. • A WSCG node may consist of one or more pipelines. Op1 Op2 Op3 Op4 Op5
- 22. A Single Pipeline in WSCG • A string of non-blocking operators form a pipeline in WSCG • The head/source: - Implement doProduce() - the driving loop producing source data. • The rest: - doProduce() - fall through to head of the pipeline. - Implement doConsume() for its own processing logic. Op1 Op2 Op3 WSCG consume consume consume produceproduceproduce produce Generate Code
- 23. A Single Pipeline Example while (table.hasNext()) { InternalRow row = table.next(); if (shouldStop()) return; } Scan Filter Project Generated for RowIterator WholeStageCodegen SELECT sid FROM emps WHERE age < 36 START: produce produce produce produce
- 24. A Single Pipeline Example while (table.hasNext()) { InternalRow row = table.next(); if (row.getInt(2) < 36) { } if (shouldStop()) return; } Scan Filter Project Generated for RowIterator WholeStageCodegen SELECT sid FROM emps WHERE age < 36 START: produce consume produce produce produce
- 25. A Single Pipeline Example while (table.hasNext()) { InternalRow row = table.next(); if (row.getInt(2) < 36) { String sid = row.getString(0); rowWriter.write(0, sid); } if (shouldStop()) return; } Scan Filter Project Generated for RowIterator WholeStageCodegen SELECT sid FROM emps WHERE age < 36 START: produce consume consume produce produce produce
- 26. A Single Pipeline Example while (table.hasNext()) { InternalRow row = table.next(); if (row.getInt(2) < 36) { String sid = row.getString(0); rowWriter.write(0, sid); ret = rowWriter.getRow(); } if (shouldStop()) return; } Scan Filter Project Generated for RowIterator WholeStageCodegen SELECT sid FROM emps WHERE age < 36 START: produce consume consume consumeproduce produce produce
- 27. • Head (source) operator: - The source, w/ or w/o input RDDs - e.g., Scan, SortMergeJoin • Non-blocking operators: - In the middle of the pipeline - e.g., Filter, Project Multiple Pipelines in WSCG • End (sink): RowIterator - Pulls result from the last pipeline • Blocking operators: - End of the previous pipeline - Start of a new pipeline - e.g., HashAggregate, Sort WSCG Pipeline 2Pipeline 1 Op1 Op2 Op3 Op4 Op5 non-blockingsource non-blockingblocking RowIterator sink
- 28. Blocking Operators in WSCG • A Blocking operator, e.g., HashAggregateExec, SortExec, break pipelines, so there may be multiple pipelines in one WSCG node. • A Blocking operator’s doConsume(): - Implement the callback to build intermediate result. • A Blocking operator’s doProduce(): - Consume the entire output from upstream to finish building the intermediate result. - Start a new loop and produce output for downstream based on the intermediate result.
- 29. A Blocking Operator Example - HashAgg while (table.hasNext()) { InternalRow row = table.next(); int age = row.getInt(2); hashMap.insertOrIncrement(sid); } HashAggregate doProduce() child.produce() HashAggregate Scan WholeStageCodegen SELECT age, count(*) FROM emps GROUP BY age consume START: produce produce
- 30. A Blocking Operator Example - HashAgg while (table.hasNext()) { InternalRow row = table.next(); int age = row.getInt(2); hashMap.insertOrIncrement(sid); } while (hashMapIter.hasNext()) { Entry e = hashMapIter.next(); rowWriter.write(0, e.getKey()); rowWriter.write(1, e.getValue()); ret = rowWriter.getRow(); if (shouldStop()) return; } HashAggregate doProduce() child.produce() HashAggregate Scan start a new pipeline WholeStageCodegen SELECT age, count(*) FROM emps GROUP BY age consume consume START: produce produce
- 31. • BHJ (broadcast-hash-join) is a pipelined operator. • BHJ executes the build side job first, the same way as in non-WSCG. • BHJ is fused together with the probe side plan (i.e., streaming plan) in WSCG. WSCG: BHJ vs. SMJ Job 2 WSCG Scan A BroadcastHashJoin WSCG HashAggregate ShuffleExchange HashAggregate Job 1 WSCG Scan B Filter BroadcastExchange
- 32. Job 1 • SMJ (sort-merge-join) is NOT fused with either child plan for WSCG. Child plans are separate WSCG nodes. • Thus, SMJ must be the head operator of a WSCG node. WSCG: BHJ vs. SMJ WSCG SortMergeJoin WSCG HashAggregate ShuffleExchange HashAggregate WSCG Scan B Filter WSCG Scan A WSCG Sort ShuffleExchange WSCG Sort ShuffleExchange
- 33. WSCG Limitations • Problems: - No JIT compilation for bytecode size over 8000 bytes (*). - Over 64KB methods NOT allowed by Java Class format. • Solutions: - Fallback - spark.sql.codegen.fallback; spark.sql.codegen.hugeMethodLimit - Move blocking loops into separate methods, e.g. hash-map building in HashAgg and sort buffer building in Sort. - Split consume() into individual methods for each operator - spark.sql.codegen.splitConsumeFuncByOperator
- 34. About Us Software Engineers • Maryann Xue PMC of Apache Calcite & Apache Phoenix @maryannxue • Xingbo Jiang Apache Spark Committer @jiangxb1987 • Kris Mok OpenJDK Committer @rednaxelafx 34
- 36. A Physical Plan Example SELECT a1, sum(b1) FROM A JOIN B ON A.key = B.key WHERE b1 < 1000 GROUP BY a1 Scan A Scan B Filter BroadcastExchange BroadcastHashJoin HashAggregate ShuffleExchange HashAggregate
- 37. RDD and Partitions RDD(Resilient Distributed Dataset) represents an immutable, partitioned collection of elements that can be operated in parallel. Partition Partition Partition Node1 Node2 Node3 RDD
- 38. Physical Operator Filter Volcano iterator model while (iter.hasNext()) { val tmpVal = iter.next() if (condition(tmpVal)) { return tmpVal } } output Partition Partition Partition RDD
- 39. Job 2 Stage 1 Stage 2 A Physical Plan Example - Scheduling Scan A BroadcastHashJoin HashAggregate ShuffleExchange HashAggregate Job 1 Stage 1 Scan B Filter BroadcastExchange
- 40. Stage 1 Stage Execution TaskSet0 Scan A BroadcastHashJoin HashAggregate Partition0 Partition1 Partition2 Partition3
- 43. How to run a Task Executor(spark.executor.cores=5) Task0 spark.task.cpus=1 Task2 Task6 Task1 Task5 Task7 Task3 Task4
- 44. Fault Tolerance ● MPP-like analytics engines(e.g., Teradata, Presto, Impala): ○ Coarser-grained recovery model ○ Retry an entire query if any machine fails ○ Short/simple queries ● Spark SQL: ○ Mid-query recovery model ○ RDDs track the series of transformations used to build them (the lineage) to recompute lost partitions. ○ Long/complex queries [e.g., complex UDFs]
- 45. Handling Task Failures Task Failure ● Record the failure count of the task ● Retry the task if failure count < maxTaskFailures ● Abort the stage and corresponding jobs if count >= maxTaskFailures Fetch Failure ● Don’t count the failure into task failure count ● Retry the stage if stage failure < maxStageFailures ● Abort the stage and corresponding jobs if stage failure >= maxStageFailures ● Mark executor/host as lost (optional)
- 47. Memory Consumption in Executor JVM Challenges: • Task run in a shared-memory environment. • Memory resource is not enough! Spark uses memory for: • RDD Storage [e.g., call cache()]. • Execution memory [e.g., Shuffle and aggregation buffers] • User code [e.g., allocate large arrays]
- 48. Execution Memory • Buffer intermediate results • Normally short lived Execution Memory Storage Memory User Memory Reserved Memory
- 49. Storage Memory • Reuse data for future computation • Cached data can be long-lived • LRU eviction for spill data Execution Memory Storage Memory User Memory Reserved Memory
- 50. Unified Memory Manager • Express execution and storage memory as one single unified region • Keep acquiring execution memory and evict storage as you need more execution memory Execution Memory (1.0 - spark.memory.storageFraction) * USABLE_MEMORY Storage Memory spark.memory.storageFraction * USABLE_MEMORY User Memory (1.0 - spark.memory.fraction) * (SYSTEM_MEMORY - RESERVED_MEMORY) Reserved Memory RESERVED_SYSTEM_MEMORY_BYTES (300MB)
- 51. Dynamic occupancy mechanism spark.memory.storageFraction • If one of its space is insufficient but the other is free, then it will borrow the other’s space. • If both parties don’t have enough space, evict storage memory using LRU mechanism.
- 52. One problem remains... • The memory resource is not enough! On-Heap Memory Off-Heap Memory Inside JVM Managed by GC Outside JVM Not managed by GC Executor Process
- 53. Off-Heap Memory • Enabled by spark.memory.offHeap.enabled • Memory size controlled by spark.memory.offHeap.size Execution Memory Storage Memory
- 54. Off-Heap Memory • Pros • Speed: Off-Heap Memory > Disk • Not bound by GC • Cons • Manually manage memory allocation/release
- 55. Tuning Data Structures In Spark applications: • Prefer arrays of objects instead of collection classes (e.g., HashMap) • Avoid nested structures with a lot of small objects and pointers when possible • Use numeric IDs or enumeration objects instead of strings for keys
- 56. Tuning Memory Config spark.memory.fraction • More execution and storage memory • Higher risk of OOM spark.memory.storageFraction • Increase storage memory to cache more data • Less execution memory may lead to tasks spill more often
- 57. Tuning Memory Config spark.memory.offHeap.enabled spark.memory.offHeap.size • Off-Heap memory not bound by GC • On-Heap + Off-Heap memory must fit in total executor memory (spark.executor.memory) spark.shuffle.file.buffer spark.unsafe.sorter.spill.reader.buffer.size • Buffer shuffle file to amortize disk I/O • More execution memory consumption
- 58. About Us Software Engineers • Maryann Xue PMC of Apache Calcite & Apache Phoenix @maryannxue • Xingbo Jiang Apache Spark Committer @jiangxb1987 • Kris Mok OpenJDK Committer @rednaxelafx 58
- 60. Vectorized Readers Read columnar format data as-is without converting to row format. • Apache Parquet • Apache ORC • Apache Arrow • ... 60
- 61. Vectorized Readers Parquet vectorized reader is 9 times faster than the non- vectorized one. See blog post 61
- 62. Vectorized Readers Supported built-in data sources: • Parquet • ORC Arrow is used for intermediate data in PySpark. 62
- 63. Implement DataSource DataSource v2 API provides the way to implement your own vectorized reader. • PartitionReaderFactory • supportColumnarReads(...) to return true • createColumnarReader(...) to return PartitionReader[ColumnarBatch] • [SPARK-25186] Stabilize Data Source V2 API 63
- 64. Delta Lake • Full ACID transactions • Schema management • Scalable metadata handling • Data versioning and time travel • Unified batch/streaming support • Record update and deletion • Data expectation Delta Lake: https://delta.io/ Documentation: https://docs.delta.io For details, refer to the blog https://tinyurl.com/yxhbe2lg
- 65. Agenda 65 UDF
- 66. What’s behind foo(x) in Spark SQL? What looks like a function call can be a lot of things: • upper(str): Built-in function • max(val): Aggregate function • max(val) over …: Window function • explode(arr): Generator • myudf(x): User-defined function • myudaf(x): User-defined aggregate function • transform(arr, x -> x + 1): Higher-order function • range(10): Table-value function
- 67. Functions in Spark SQL Builtin Scalar Function Java / Scala UDF Python UDF (*) Aggregate / Window Higher-order Function Scope 1 Row 1 Row 1 Row Whole table 1 Row Data Feed Scalar expressions Scalar expressions Batch of data Scalar expressions + aggregate buffer Expression of complex type Process Same JVM Same JVM Python Worker process Same JVM Same JVM Impl. Level Expression Expression Physical Operator Physical Operator Expression Data Type Internal External External Internal Internal (*): and all other non-Java user-defined functions
- 68. UDF execution User Defined Functions: • Java/Scala UDFs • Hive UDFs • when Hive support enabled Also we have: • Python/Pandas UDFs • will talk later in PySpark execution 68
- 69. Java/Scala UDFs • UDF: User Defined Function • Java/Scala lambdas or method references can be used. • UDAF: User Defined Aggregate Function • Need to implement UserDefinedAggregateFunction. 69
- 70. UDAF Implement UserDefinedAggregateFunction • def initialize(...) • def update(...) • def merge(...) • def evaluate(...) • ... 70
- 71. Hive UDFs Available when Hive support enabled. • Register using create function command • Use in HiveQL 71
- 72. Hive UDFs Provides wrapper expressions for each UDF type: • HiveSimpleUDF: UDF • HiveGenericUDF: GenericUDF • HiveUDAFFunction: UDAF • HiveGenericUDTF: GenericUDTF 72
- 73. UDF execution 1. Before invoking UDFs, convert arguments from internal data format to objects suitable for each UDF types. • Java/Scala UDF: Java/Scala objects • Hive UDF: ObjectInspector 2. Invoke the UDF. 3. After invocation, convert the returned values back to internal data format. 73
- 75. PySpark PySpark is a set of Python bindings for Spark APIs. • RDD • DataFrame • other libraries based on RDDs, DataFrames. • MLlib, Structured Streaming, ... Also, SparkR: R bindings for Spark APIs 75
- 76. PySpark RDD vs. DataFrame: • RDD invokes Python functions on Python worker • DataFrame just constructs queries, and executes it on the JVM. • except for Python/Pandas UDFs 76
- 77. PySpark execution Python script drives Spark on JVM via Py4J. Executors run Python worker. 77 Python Driver Executor Python Worker Executor Python Worker Executor Python Worker
- 78. PySpark and Pandas Ease of interop: PySpark can convert data between PySpark DataFrame and Pandas DataFrame. • pdf = df.toPandas() • df = spark.createDataFrame(pdf) 78 Note: df.toPandas() triggers the execution of the PySpark DataFrame, similar to df.collect()
- 79. PySpark and Pandas (cont’d) New way of interop: Koalas brings the Pandas API to Apache Spark 79 import databricks.koalas as ks import pandas as pd pdf = pd.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':['a','b','b']}) # Create a Koalas DataFrame from pandas DataFrame df = ks.from_pandas(pdf) # Rename the columns df.columns = ['x', 'y', 'z1'] # Do some operations in place: df['x2'] = df.x * df.x https://github.com/databricks/koalas
- 81. Python UDF and Pandas UDF @udf('double') def plus_one(v): return v + 1 @pandas_udf('double', PandasUDFType.SCALAR) def pandas_plus_one(vs): return vs + 1 81
- 82. Python/Pandas UDF execution 82 Invoke UDFPythonRunner PhysicalOperator Batch of data Deserializer SerializerBatch of data
- 83. Python UDF execution 83 Invoke UDFPythonUDFRunner PhysicalOperator Batch of Rows Batch of Rows Deserializer Serializer
- 84. Pandas UDF execution 84 Invoke UDFArrowPythonRunner PhysicalOperator Batch of Columns Batch of Columns Deserializer Serializer
- 85. Python/Pandas UDFs Python UDF • Serialize/Deserialize data with Pickle • Fetch data in blocks, but invoke UDF row by row Pandas UDF • Serialize/Deserialize data with Arrow • Fetch data in blocks, and invoke UDF block by block 85
- 86. Python/Pandas UDFs Pandas UDF perform much better than row-at-a-time Python UDFs. • 3x to over 100x See blog post 86
- 87. Further Reading This Spark+AI Summit: • Understanding Query Plans and Spark Previous Spark Summits: • A Deep Dive into Spark SQL’s Catalyst Optimizer • Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal • Improving Python and Spark Performance and Interoperability with Apache Arrow
- 88. Thank you Maryann Xue (maryann.xue@databricks.com) Xingbo Jiang (xingbo.jiang@databricks.com) Kris Mok (kris.mok@databricks.com) 88