Presented at Hadoop Summit 2013 Hive User Group Meetup

1. © Hortonworks Inc. 2011
Hive Correlation Optimizer
Yin Huai
yhuai@hortonworks.com
huai@cse.ohio-state.edu
Page 1
Hadoop Summit 2013 Hive User Group Meetup

2. © Hortonworks Inc. 2011
About me
•Hive contributor
•Summer intern at Hortonworks
•4th year Ph.D. student at The Ohio State
University
•Research interests: query optimizations, file
formats, distributed systems, and storage
systems
Page 2
Architecting the Future of Big Data

3. © Hortonworks Inc. 2011
Outline
•Query planning in Hive
•Correlations in a query (Intra-query
correlations)
•Case studies
•Automatically exploiting correlations (HIVE-
2206: Correlation Optimizer)
Page 3
Architecting the Future of Big Data

4. © Hortonworks Inc. 2011
Query planning
Page 4
Architecting the Future of Big Data
SELECT t1.c2, count(*)
FROM t1 JOIN t2 ON (t1.c1=t2.c1)
GROUP BY t1.c2
t1 t2
JOIN
AGG
t1.c1=t2.c1
Calculate count(*) for
every group of t1.c2

5. © Hortonworks Inc. 2011
Query planning
Page 5
Architecting the Future of Big Data
SELECT t1.c2, count(*)
FROM t1 JOIN t2 ON (t1.c1=t2.c1)
GROUP BY t1.c2
t1 t2
JOIN
AGG Evaluate this query in
distributed systems
t1 t2
JOIN
AGG
Shuffle
Shuffle
c1
c2
How to shuffle?
Use the key column(s)

6. © Hortonworks Inc. 2011
Generating MapReduce jobs
Page 6
Architecting the Future of Big Data
t1 t2
JOIN
AGG
Shuffle
Shuffle c2
c1
t1 t2
JOIN
Shuffle
tmp
c1
tmp
AGG
Shuffle c2
1 MR job can shuffle
data once
Job 1
Job 2

7. © Hortonworks Inc. 2011
Generating MapReduce jobs
Page 7
Architecting the Future of Big Data
t1 t2
JOIN
Shuffle
tmp
c1
tmp
AGG
Shuffle c2
MapReuce will shuffle
data for us, we just
need to emit outputs
from the Map phase
We use ReduceSinkOperator
(RS) to emit Map outputs.
RSs are the end of a Map phase.
t1 t2
JOIN
tmp
tmp
AGG
RS1 RS2
RS2
Job 1
Map
Job 1
Reduce
Job 2
Map
Job 2
Reduce

8. © Hortonworks Inc. 2011
Outline
•Query planning in Hive
•Correlations in a query (Intra-query
correlations)
•Case studies
•Automatically exploiting correlations (HIVE-
2206: Correlation Optimizer)
Page 8
Architecting the Future of Big Data

9. © Hortonworks Inc. 2011
Intra-query correlations
Page 9
Architecting the Future of Big Data
SELECT x.c1, count(*)
FROM t1 x JOIN t1 y ON (x.c1=y.c1)
GROUP BY x.c1
t1 as x t1 as y
JOIN
AGG
x.c1=y.c1
Calculate count(*) for
every group of x.c1
Correlations:
1. Same input tables
2. JOIN and AGG using the
same key

10. © Hortonworks Inc. 2011
Intra-query correlations
Page 10
Architecting the Future of Big Data
x.c1=y.c1
Calculate count(*)
for every group of
z.c1
t1 as x t2 as y
JOIN1
JOIN2
AGG1
t1 as z
p.c1=q.c1
SELECT p.c1, q.c2, q.cnt
FROM (SELECT x.c1 AS c1 FROM t1 x JOIN t2 y ON (x.c1=y.c1)) p
JOIN (SELECT z.c1 AS c1, count(*) AS cnt FROM t1 z GROUP BY z.c1) q
ON (p.c1=q.c1)
Correlations:
1. Same input tables (t1)
2. JOIN1 and AGG1 using the
same key
3. JOIN2 and all of its parents
using the same key

11. © Hortonworks Inc. 2011
Intra-query correlations
• Defined in “YSmart: Yet Another SQL-to-MapReduce Translator”
– http://ysmart.cse.ohio-state.edu/
– http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-11-7.pdf
• Targeting on operators which need to shuffle the data and inputs
• Three kinds of correlations
– Input correlation (IC): independent operators share the same input tables
– Transit correlation (TC): independent operators have input correlation and
also shuffle the data in the same way (e.g. using the same keys)
– Job flow correlation (JFC): two dependent operators shuffle the data in
the same way
Page 11
Architecting the Future of Big Data
t1 as x t2 as y
JOIN1 AGG1
t1 as z
IC
t1 as x t2 as y
JOIN1 AGG1
t1 as z
x.c1=y.c1 group by z.c1
TC
JOIN
AGG
x.c1=y.c1
group by z.c1
JFC

12. © Hortonworks Inc. 2011
Correlation-unaware query planning
Page 12
Architecting the Future of Big Data
t1 t1
JOIN
AGG
Shuffle
Shuffle c1
c1
Hive does not care:
1. If a table has been
used multiple
times
2. If data really needs
to be shuffled
t1 t1
JOIN
Shuffle
tmp
c1
Job 1
tmp
AGG
Shuffle c1 Job 2
Drawbacks:
1. Unnecessary data
loading
2. Unnecessary data
shuffling
3. Unnecessary data
materialization

13. © Hortonworks Inc. 2011
Outline
•Query planning in Hive
•Correlations in a query (Intra-query
correlations)
•Case studies
•Automatically exploiting correlations (HIVE-
2206: Correlation Optimizer)
Page 13
Architecting the Future of Big Data

14. © Hortonworks Inc. 2011
Case studies: TPC-H Q17 (Flattened)
SELECT
sum(l_extendedprice) / 7.0 as avg_yearly
FROM
(SELECT l_partkey, l_quantity, l_extendedprice
FROM lineitem JOIN part ON (p_partkey=l_partkey)
WHERE p_brand='Brand#35’ AND
p_container = 'MED PKG’) touter
JOIN
(SELECT l_partkey as lp, 0.2 * avg(l_quantity) as lq
FROM lineitem
GROUP BY l_partkey) tinner
ON (touter.l_partkey = tinnter.lp)
WHERE touter.l_quantity < tinner.lq
Page 14
Architecting the Future of Big Data

15. © Hortonworks Inc. 2011
Case studies: TPC-H Q17 (Flattened)
Page 15
Architecting the Future of Big Data
lineitem part
JOIN1
JOIN2
AGG1
lineitem
AGG2
lineitem is used by JOIN1 and AGG1
JOIN1, AGG1, and JOIN2 share the same key

16. © Hortonworks Inc. 2011
Case studies: TPC-H Q17 (Flattened)
Page 16
Architecting the Future of Big Data
lineitem part
JOIN1
JOIN2
AGG1
lineitem
AGG2
Job 1 Job 2
Job 3
Job 4
Without Correlation Optimizer

17. © Hortonworks Inc. 2011
Case studies: TPC-H Q17 (Flattened)
Page 17
Architecting the Future of Big Data
lineitem part
JOIN1
JOIN2
AGG1
lineitem
AGG2
part
JOIN1
JOIN2
AGG1
lineitem
AGG2
Job 1 Job 2
Job 3
Job 4 Job 2
Job 1
Without Correlation Optimizer With Correlation Optimizer

18. © Hortonworks Inc. 2011
Case studies: TPC-DS Q95 (Flattened)
SELECT count(distinct ws1.ws_order_number) as order_count,
sum(ws1.ws_ext_ship_cost) as total_shipping_cost,
sum(ws1.ws_net_profit) as total_net_profit
FROM web_sales ws1
JOIN customer_address ca ON (ws1.ws_ship_addr_sk = ca.ca_address_sk)
JOIN web_site s ON (ws1.ws_web_site_sk = s.web_site_sk)
JOIN date_dim d ON (ws1.ws_ship_date_sk = d.d_date_sk)
LEFT SEMI JOIN (SELECT ws2.ws_order_number as ws_order_number
FROM web_sales ws2 JOIN web_sales ws3
ON(ws2.ws_order_number = ws3.ws_order_number)
WHERE ws2.ws_warehouse_sk <> ws3.ws_warehouse_sk) ws_wh1
ON (ws1.ws_order_number = ws_wh1.ws_order_number)
LEFT SEMI JOIN (SELECT wr_order_number
FROM web_returns wr
JOIN (SELECT ws4.ws_order_number as ws_order_number
FROM web_sales ws4 JOIN web_sales ws5
ON (ws4.ws_order_number = ws5.ws_order_number)
WHERE ws4.ws_warehouse_sk <> ws5.ws_warehouse_sk) ws_wh2
ON (wr.wr_order_number = ws_wh2.ws_order_number)) tmp1
ON (ws1.ws_order_number = tmp1.wr_order_number)
WHERE d.d_date >= '2001-05-01' AND
d.d_date <= '2001-06-30’ AND
ca.ca_state = 'NC’ AND
s.web_company_name = 'pri'
Page 18
Architecting the Future of Big Data

19. © Hortonworks Inc. 2011
Case studies: TPC-DS Q95 (Flattened)
Page 19
Architecting the Future of Big Data
web_sales
AGG
customer_address web_site
Map
Join
Semi
Join
web_sales web_sales
JOIN1
web_sales web_sales
JOIN1
web_returns
JOIN2
date_dim

20. © Hortonworks Inc. 2011
Case studies: TPC-DS Q95 (Flattened)
Page 20
Architecting the Future of Big Data
web_sales
AGG
customer_address web_site
Map
Join
Semi
Join
web_sales web_sales
JOIN1
web_sales web_sales
JOIN1
web_returns
JOIN2
Without Correlation Optimizer
• 6 MapReduce jobs
• Unnecessary data loading
(black web_sales nodes)
• Unnecessary data shuffling
Job 6
Job 2
Job 3
Job 4
Job 5
Job 1
date_dim

21. © Hortonworks Inc. 2011
Case studies: TPC-DS Q95 (Flattened)
Page 21
Architecting the Future of Big Data
web_sales
AGG
customer_address web_site
Map
Join
Semi
Join
web_sales
JOIN1
JOIN1
web_returns
JOIN2
With Correlation Optimizer
• Black web_sales nodes share
the same data loading
date_dim

22. © Hortonworks Inc. 2011
Case studies: TPC-DS Q95 (Flattened)
Page 22
Architecting the Future of Big Data
web_sales
AGG
customer_address web_site
Map
Join
Semi
Join
web_sales
JOIN1
JOIN1
web_returns
JOIN2
With Correlation Optimizer
• Black web_sales nodes share
the same data loading
• 3 MapReduce jobs
Job 1
Job 2
Job 3
date_dim

23. © Hortonworks Inc. 2011
Case studies: TPC-DS Q95 (Flattened)
Page 23
Architecting the Future of Big Data
web_sales
AGG
customer_address web_site
Map
Join
Semi
Join
web_sales
JOIN1
web_returns
JOIN2
Follow-up work
• Evaluate JOIN1 only once
without materializing a
temporary table
date_dim

24. © Hortonworks Inc. 2011
Case studies: TPC-DS Q95 (Flattened)
Page 24
Architecting the Future of Big Data
web_sales
AGG
customer_address web_site
Map
Join
Semi
Join
web_sales
JOIN1
web_returns
JOIN2
Follow-up work
• Evaluate JOIN1 only once
without materializing a
temporary table
• Only use 2 MapReduce jobs
Job 1
Job 2
date_dim

25. © Hortonworks Inc. 2011
Outline
•Query planning in Hive
•Correlations in a query (Intra-query
correlations)
•Case studies
•Automatically exploiting correlations (HIVE-
2206: Correlation Optimizer)
Page 25
Architecting the Future of Big Data

26. © Hortonworks Inc. 2011
Objectives
• Eliminate unnecessary data loading
– Query planner will be aware what data will be loaded
– Do as many things as possible for loaded data
• Eliminate unnecessary data shuffling
– Query planner will be aware when data really needs to be shuffled
– Do as many things as possible before shuffling the data again
Page 26
Architecting the Future of Big Data

27. © Hortonworks Inc. 2011
ReduceSink Deduplication
• HIVE-2340
• Handle chained Job Flow Correlations
– e.g. Generating a single job for both Group By and Order By
• Cannot handle complex patterns
– e.g. Multiple Joins involved patterns
• Need a fundamental solution
• Need to exploit shared input tables
Page 27
Architecting the Future of Big Data
t1
RS1
AGG1
RS2
…
t1
RS1
AGG1
…

28. © Hortonworks Inc. 2011
Correlation Optimizer
• 2-phase optimizer
– Phase 1: Correlation Detection
– Phase 2: Query plan tree transformation
• This work is not just about the optimizer
– New operators to support the execution of an optimized plan
– A mechanism to coordinate the operator tree inside the Reduce phase
Page 28
Architecting the Future of Big Data

29. © Hortonworks Inc. 2011
Correlation detection
Page 29
Architecting the Future of Big Data
SELECT p.c1, q.c2, q.cnt
FROM (SELECT x.c1 AS c1 FROM t1 x JOIN t2 y ON (x.c1=y.c1)) p
JOIN (SELECT z.c1 AS c1, count(*) AS cnt FROM t1 z GROUP BY z.c1) q
ON (p.c1=q.c1)
1. Traverse the tree all the way
down to find matching keys
in ReduceSinkOperators
2. Then, check input tables to
find shared data loading
opportunities
t1 as x t2 as y
JOIN1
JOIN2
AGG1
t1 as z
RS1 RS2 RS3
RS4 RS5
Key: p.c1 Key: q.c1
Key: x.c1 Key: y.c1 Key: z.c1

30. © Hortonworks Inc. 2011
Query plan tree transformation
Page 30
Architecting the Future of Big Data
SELECT p.c1, q.c2, q.cnt
FROM (SELECT x.c1 AS c1 FROM t1 x JOIN t2 y ON (x.c1=y.c1)) p
JOIN (SELECT z.c1 AS c1, count(*) AS cnt FROM t1 z GROUP BY z.c1) q
ON (p.c1=q.c1)
t1 as x t2 as y
JOIN1
JOIN2
AGG1
t1 as z
Key: p.c1
RS1 RS2 RS3
RS4 RS5
Key: q.c1
Key: x.c1 Key: y.c1 Key: z.c1
t1 as x, zt2 as y
JOIN1
JOIN2
AGG1
RS1RS2 RS3

31. © Hortonworks Inc. 2011
Thanks
Architecting the Future of Big Data
Page 31