2015-01-14 道玄坂LT祭り(ミドル・インフラ) in Japan 『Presto + MySQLで分散SQL』 by Sadayuki Furuhashi

1. Sadayuki Furuhashi
Founder & Software Architect
Treasure Data, inc.
Presto + MySQL
道玄坂LT祭り
で分散SQL

2. A little about me...
> Sadayuki Furuhashi
> github/twitter: @frsyuki
> Treasure Data, Inc.
> Founder & Software Architect
> Open-source hacker
> MessagePack - Efficient object serializer
> Fluentd - An unified data collection tool
> ServerEngine - A Ruby framework to build multiprocess servers
> Prestogres - PostgreSQL protocol gateway for Presto
> LS4 - A distributed object storage with cross-region replication
> kumofs - A distributed strong-consistent key-value data store

3. Check: www.treasuredata.com
Cloud service for the entire data pipeline,
including Presto. We’re hiring!

6. What’s Presto?
A distributed SQL query engine
for interactive data analisys
against GBs to PBs of data.

7. Client
Coordinator Connector
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service

8. Client
Coordinator Connector
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service
1. find servers in a cluster

9. Client
Coordinator Connector
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service
2. Client sends a query
using HTTP

10. Client
Coordinator Connector
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service
3. Coordinator builds
a query plan
Connector plugin
provides metadata
(table schema, etc.)

11. Client
Coordinator Connector
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service
4. Coordinator sends
tasks to workers

12. Client
Coordinator Connector
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service
5. Workers read data
through connector plugin

13. Client
Coordinator Connector
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service
6. Workers run tasks
in memory

14. Coordinator Connector
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service
7. Client gets the result
from a worker
Client

15. Client
Coordinator Connector
Plugin
Worker
Worker
Worker
Storage / Metadata
Discovery Service

16. Client
Coordinator Hive
Connector
Worker
Worker
Worker
HDFS,
Hive Metastore
Discovery Service
find servers in a cluster
Hive connector

17. Client
Coordinator JDBC
Connector
Worker
Worker
Worker
Cassandra
Discovery Service
find servers in a cluster
Cassandra connector

18. Client
Coordinator
other
connectors
...
Worker
Worker
Worker
PostgreSQL
Discovery Service
find servers in a cluster
Hive
Connector
HDFS / Metastore
Multiple connectors in a query
JDBC
Connector
Other data sources...

19. All stages are pipe-lined
✓ No wait time
✓ No fault-tolerance
MapReduce vs. Presto
MapReduce Presto
map map
reduce reduce
task task
task task
task
task
memory-to-memory
data transfer
✓ No disk IO
✓ Data chunk must
fit in memory
task
disk
map map
reduce reduce
disk
disk
Write data
to disk
Wait between
stages

20. Presto meetup!

21. Presto
JOIN
Hive
MySQL
client
select orderkey, orderdate, custkey, email
from orders
join mysql.presto_test.users
on orders.custkey = users.id
order by custkey, orderdate;

22. Presto
JOIN
Hive
MySQLINSERT INTO
client
create table mysql.presto_test.recent_user_info
as
select users.id, users.email, count(1) as count
from orders
join mysql.presto_test.users
on orders.custkey = users.id
group by 1, 2;

23. Presto
JOIN
Hive
MySQL
$ psql Prestogres

24. Presto
JOIN
Hive
MySQL
$ psql Prestogres
PostgreSQL protocol gateway
for Presto