Druid is an open-source analytics data store specially designed to execute OLAP queries on event data. Its speed, scalability and efficiency have made it a popular choice to power user-facing analytic applications, including multiple BI tools and dashboards. However, Druid does not provide important features requested by many of these applications, such as a SQL interface or support for complex operations such as joins. This talk presents our work on extending Druid indexing and querying capabilities using Apache Hive. In particular, our solution allows to index complex query results in Druid using Hive, query Druid data sources from Hive using SQL, and execute complex Hive queries on top of Druid data sources. We describe how we built an extension that brings benefits to both systems alike, leveraging Apache Calcite to overcome the challenge of transparently generating Druid JSON queries from the input Hive SQL queries. We conclude with a demo highlighting the performant and powerful integration of these projects.

1. Interactive Analytics at Scale
in Apache Hive using Druid
Jesús Camacho Rodríguez
DataWorks Summit Europe
April 5, 2017

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Motivation
 BI/OLAP applications that require interactive
visualization of complex data streams
– Real time bidding events
– User activity streams
– Voice call logs
– Network traffic flows
– Firewall events
– Application performance metrics
 Querying event data at large scale poses multiple challenges
Interactive analytics on event data

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid overview
 Development starts in 2011, open-sourced in late 2012
 Initial use case: interactive ad-analytics
 +150 contributors
 Main features
– Column-oriented distributed data store
– Batch and real-time ingestion
– Scalable to petabytes of data
– Sub-second response for arbitrary time-based
slice-and-dice
• Data partitioned by time dimension
• Automatic data summarization
• Approximate algorithms (hyperLogLog, theta)
Most Events per Day
30 Billion Events / Day
(Metamarkets)
Most Computed Metrics
1 Billion Metrics / Min
(Jolata)
Largest Cluster
200 Nodes
(Metamarkets)
Largest Hourly Ingestion
2TB per Hour
(Netflix)

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid architecture
Dashboards, BI tools

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Persistent storage
 Data in Druid is stored in segment files
 Partitioned by time, supports fast time-based slice-and-dice
 Ideally, segment files are each smaller than 1GB
 If files are large, smaller time partitions are needed
Time
Segment 1:
Monday
Segment 2:
Tuesday
Segment 3:
Wednesday
Segment 4:
Thursday
Segment 5:
Friday

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Segment data structures
 Within a segment
– Timestamp column
– Dimension columns
– Metric columns
– Indexes to facilitate fast lookup and aggregation

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Querying
 HTTP REST API
 Queries and results expressed in JSON
 Multiple query types
– Time boundary
– Segment metadata
– Timeseries
– TopN
– GroupBy
– Select
{
"queryType": "groupBy",
"dataSource": "product_sales_index",
"granularity": "all",
"dimension": "product_id",
"aggregations": [ { "type": "doubleSum", "name": "s", "fieldName": "sales" } ],
"limitSpec": {
"limit": 10,
"columns": [ {"dimension": "s", "direction": "descending" } ]
},
"intervals": [ "2010-01-01T00:00:00.000/2012-01-01T00:00:00.000" ]
}
Important to use adequate type  Impact on query performance

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid + Apache Hive
 Integration brings benefits both to Druid and Apache Hive
– Indexing complex query results in Druid using Hive
– Introducing a SQL interface on top of Druid
– Being able to execute complex operations on Druid data
– Efficient execution of OLAP queries in Hive

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Interactive Analytics at Scale in Hive using Druid
Introduction
Registering and creating Druid data sources

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid data sources in Hive
 User needs to provide Druid data sources information to Hive
 Two different options depending on requirements
– Register Druid data sources in Hive
• Data is already stored in Druid
– Create Druid data sources from Hive
• Data is stored in Hive
• User may want to pre-process the data before storing it in Druid

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid data sources in Hive
 Simple CREATE EXTERNAL TABLE statement
CREATE EXTERNAL TABLE druid_table_1
STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
TBLPROPERTIES ("druid.datasource" = "wikiticker");
Hive table name
Hive storage handler classname
Druid data source name
⇢ Broker node endpoint specified as a Hive configuration parameter
⇢ Automatic Druid data schema discovery: segment metadata query
Registering Druid data sources

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid data sources in Hive
 Use Create Table As Select (CTAS) statement
CREATE TABLE druid_table_1
STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
TBLPROPERTIES ("druid.datasource" = "wikiticker", "druid.segment.granularity" = "DAY")
AS
SELECT __time, page, user, c_added, c_removed
FROM src;
Hive table name
Hive storage handler classname
Druid data source name
Druid segment granularity
Creating Druid data sources

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid data sources in Hive
 Use Create Table As Select (CTAS) statement
CREATE TABLE druid_table_1
STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler’
TBLPROPERTIES ("druid.datasource" = "wikiticker", "druid.segment.granularity" = "DAY”)
AS
SELECT __time, page, user, c_added, c_removed
FROM src;
⇢ Inference of Druid column types (timestamp, dimensions, metrics) depends on Hive column type
Creating Druid data sources
Timestamp Dimensions Metrics

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid data sources in Hive
 File Sink operator uses Druid output format
– Creates segment files and register them in Druid
– Data needs to be partitioned by time granularity
• Granularity specified as configuration parameter
Creating Druid data sources
Select
File Sink
Original CTAS
physical plan
__time page user c_added c_removed
2011-01-01T01:05:00Z Justin Boxer 1800 25
2011-01-02T19:00:00Z Justin Reach 2912 42
2011-01-01T11:00:00Z Ke$ha Xeno 1953 17
2011-01-02T13:00:00Z Ke$ha Helz 3194 170
2011-01-02T18:00:00Z Miley Ashu 2232 34
CTAS query results
Table Scan

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
__time page user c_added c_removed __time_granularity
2011-01-01T01:05:00Z Justin Boxer 1800 25 2011-01-01T00:00:00Z
2011-01-02T19:00:00Z Justin Reach 2912 42 2011-01-02T00:00:00Z
2011-01-01T11:00:00Z Ke$ha Xeno 1953 17 2011-01-01T00:00:00Z
2011-01-02T13:00:00Z Ke$ha Helz 3194 170 2011-01-02T00:00:00Z
2011-01-02T18:00:00Z Miley Ashu 2232 34 2011-01-02T00:00:00Z
Druid data sources in Hive
 File Sink operator uses Druid output format
– Creates segment files and register them in Druid
– Data needs to be partitioned by time granularity
• Granularity specified as configuration parameter
Creating Druid data sources
Select
File Sink
Rewritten CTAS
physical plan CTAS query results
Table Scan
Reduce
Truncate timestamp to day granularity

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 File Sink operator uses Druid output format
– Creates segment files and register them in Druid
– Data needs to be partitioned by time granularity
• Granularity specified as configuration parameter
2011-01-01T01:05:00Z Justin Boxer 1800 25 2011-01-01T00:00:00Z
2011-01-01T11:00:00Z Ke$ha Xeno 1953 17 2011-01-01T00:00:00Z
2011-01-02T19:00:00Z Justin Reach 2912 42 2011-01-02T00:00:00Z
2011-01-02T13:00:00Z Ke$ha Helz 3194 170 2011-01-02T00:00:00Z
2011-01-02T18:00:00Z Miley Ashu 2232 34 2011-01-02T00:00:00Z
Segment 2011-01-01
Segment 2011-01-02
Druid data sources in Hive
Creating Druid data sources
Select
File Sink
Rewritten CTAS
physical plan
Table Scan
Reduce
CTAS query results

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Interactive Analytics at Scale in Hive using Druid
Introduction
Registering and creating Druid data sources
Querying Druid data sources

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Querying Druid data sources
 Automatic rewriting when query is expressed over Druid table
– Powered by Apache Calcite
– Main challenge: identify patterns in logical plan corresponding to different kinds of Druid queries
(Timeseries, TopN, GroupBy, Select)
 Translate (sub)plan of operators into valid Druid JSON query
– Druid query is encapsulated within Hive TableScan operator
 Hive TableScan uses Druid input format
– Submits query to Druid and generates records out of the query results
 It might not be possible to push all computation to Druid
– Our contract is that the query should always be executed

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid query recognition (powered by Apache Calcite)
SELECT `user`, sum(`c_added`) AS s
FROM druid_table_1
WHERE EXTRACT(year FROM `__time`)
BETWEEN 2010 AND 2011
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Top 10 users that have added more characters
from beginning of 2010 until the end of 2011
Apache Hive - SQL query
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid query recognition (powered by Apache Calcite)
SELECT `user`, sum(`c_added`) AS s
FROM druid_table_1
WHERE EXTRACT(year FROM `__time`)
BETWEEN 2010 AND 2011
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Top 10 users that have added more characters
from beginning of 2010 until the end of 2011
Apache Hive - SQL query
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter
Possible to express filters
on time dimension using
SQL standard functions

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hive
Druid query
select
Druid query recognition (powered by Apache Calcite)
SELECT `user`, sum(`c_added`) AS s
FROM druid_table_1
WHERE EXTRACT(year FROM `__time`)
BETWEEN 2010 AND 2011
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Initially:
– Scan is executed in Druid (select query)
– Rest of the query is executed in Hive
Apache Hive - SQL query
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hive
Druid query
select
Rewriting
rule
Druid query recognition (powered by Apache Calcite)
SELECT `user`, sum(`c_added`) AS s
FROM druid_table_1
WHERE EXTRACT(year FROM `__time`)
BETWEEN 2010 AND 2011
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Rewriting rules push computation into Druid
– Need to check that operator meets some
pre-conditions before pushing it to Druid
Apache Hive - SQL query
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hive
Druid query
select
Rewriting
rule
Druid query recognition (powered by Apache Calcite)
SELECT `user`, sum(`c_added`) AS s
FROM druid_table_1
WHERE EXTRACT(year FROM `__time`)
BETWEEN 2010 AND 2011
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Rewriting rules push computation into Druid
– Need to check that operator meets some
pre-conditions before pushing it to Druid
Apache Hive - SQL query
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hive
Druid query
groupBy
Rewriting
rule
SELECT `user`, sum(`c_added`) AS s
FROM druid_table_1
WHERE EXTRACT(year FROM `__time`)
BETWEEN 2010 AND 2011
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Rewriting rules push computation into Druid
– Need to check that operator meets some
pre-conditions before pushing it to Druid
Druid query recognition (powered by Apache Calcite)
Apache Hive - SQL query
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hive
Druid query
groupBy
Rewriting
rule
SELECT `user`, sum(`c_added`) AS s
FROM druid_table_1
WHERE EXTRACT(year FROM `__time`)
BETWEEN 2010 AND 2011
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Rewriting rules push computation into Druid
– Need to check that operator meets some
pre-conditions before pushing it to Druid
Druid query recognition (powered by Apache Calcite)
Apache Hive - SQL query
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
{
"queryType": "groupBy",
"dataSource": "users_index",
"granularity": "all",
"dimension": "user",
"aggregations": [ { "type": "longSum", "name": "s", "fieldName": "c_added" } ],
"limitSpec": {
"limit": 10,
"columns": [ {"dimension": "s", "direction": "descending" } ]
},
"intervals": [ "2010-01-01T00:00:00.000/2012-01-01T00:00:00.000" ]
}
Physical plan transformation
Apache Hive
Druid query
groupBy
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter
Select
File SinkFile Sink
Table Scan
Query physical plan
Druid JSON query
Table Scan uses
Druid Input Format

27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid input format
 Submits query to Druid and generates records out of the query results
 Current version
– Timeseries, TopN, and GroupBy queries are not partitioned
– Select queries: realtime and historical nodes are contacted directly
Node
Table Scan
Record reader
…
Timeseries, TopN, GroupBy
Node
Table Scan
Record reader
…
Table Scan
Record reader
… Node
Table Scan
Record reader
…
Table Scan
Record reader
…
Select

28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Interactive Analytics at Scale in Hive using Druid
Introduction
Registering and creating Druid data sources
Querying Druid data sources
Demonstration

29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demonstration
 Implementation in Apache Hive 2.3 - Apache Hive 3.0
– Release in Q2 2017
– Relies on Druid 0.9.2 and Apache Calcite 1.12.0
 Current status (master)
– Registering, creating, overwritting and deleting Druid data sources
– Querying Druid from Hive
• Bypass broker for Druid Select queries

30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demonstration

31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Interactive Analytics at Scale in Hive using Druid
Introduction
Registering and creating Druid data sources
Querying Druid data sources
Demonstration
Road ahead

32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Road ahead
 Tighten integration between Druid and Apache Hive/Apache Calcite
– Recognize more functions  Push more computation to Druid
– Support complex column types
– Close the gap between semantics of different systems
• Time zone handling, null values
 Broader perspective
– Materialized views support in Apache Hive
• Data stored in Apache Hive
• Create materialized view in Druid
– Denormalized star schema for a certain time period
• Automatic input query rewriting over the materialized view (Apache Calcite)

33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Acknowledgments
 Apache Hive, Apache Calcite and Druid communities
– Slim Bouguerra, Julian Hyde, Nishant Bangarwa, Ashutosh Chauhan, Gunther Hagleitner, Carter
Shanklin, and many others

34. Thank You
@ApacheHive | @ApacheCalcite | @druidio
http://cwiki.apache.org/confluence/display/Hive/Druid+Integration
http://calcite.apache.org/docs/druid_adapter.html