1. Our Presto use case
and
performance test
Hironori Ogibayashi
Shin Matsuura
2. About us
● Hironori Ogibayashi(@angostura11)
● Shin Matsuura
○ IT Infrastructure team in Japanese
telecommunications carrier
○ Mainly working on middleware - test,
installation, deployment.
3. Todays Topic
● Presto use case
○ Deployment
○ Use case
○ Challenges
○ Future work
● Performance comparison between
Hive+Tez and Presto
6. Log Usage
● Systems Infrastructure team
○ Checking trends in server performance
○ Performance analysis of Oracle
Database
● Application development team
○ Improving system and business
operations.
7. Application for Oracle DB Performance Analysis
- Check existing/potential problems of
Oracle database, for certain system,
certain period.
- Utilize logs stored in HDFS. Queries
were executed on Hive.
- But, it took more than one hour to
get the result...
- (So, we migrated to Presto.)
8. Why Presto?
● Frequent use of Interactive / ad-hoc
queries.
● Of cource, faster is better.
9. Hadoop Slave
Presto Deployment
Hadoop Slave
DataNode
TaskTracker
Presto Worker
Presto
Coordinator
Hive Metastore
Application/Client
・・・
● A decicated physical machine as a
Coordinator.
● Workers run on each Hadoop slaves.
● Logs in HDFS are periodically
converted to RCfiles.
● Presto versions
○ 0.66⇒0.73⇒0.75⇒0.82
10. Deployment Effect - Elapsed time of a single query
230sec
7sec
- Elapsed time of one of
the queries issued by the
application.
- Query was run on CDH4
(MRv1) cluster.
11. Deployment and Operation
● Deployment
○ Easy;Just extract binaries in each server and modify
configuration file.
○ Automated by Ansible + yum.
● What we use in operation
○ Query history
■ Coordinator Web UI
○ Logs
■ /var/presto/data/logs/{server.log,launcher.log}
○ Metrics
■ presto-metrics(https://github.com/xerial/presto-
metrics)⇒Fluentd⇒Elasticsearch + Kibana
○ sys schema
12. Challenges
● Worker crash / hang.
○ OutOfMemory. In case of hanging, we resolve to “kill -9”.
○ We Modified the memory parameter: task.shard.max-
threads×task.max-memory < -Xmx
● At first, we set node-scheduler.include-coordinator=true.
In which case, Coordinator crashed due to heavy query.
● SQL difference from HiveQL
○ At first our Application used both Hive and Presto because we used
Presto experimentally.Hence the Application had to support both
HiveQL and Presto(ANSI SQL).
○ Now, the application no longer use Hive.
13. Future work
● Improve Coodinator’s availability.
● Security
○ Now, all queries are executed as Presto’s daemon user.
● Resource isolation between Presto and Hadoop daemons.
23. Queries
Query1: select count(*) from TestTBL
Query2: select * from TestTBL where col1 = ‘XXX’
Query3: select * from TestTBL where col1 = ‘XXX’ and col2 = ‘YYY’
Query4: select col1, count(*) from TestTBL group by col1
Query5: select col1, count(*) from TestTBL where col2 = ‘YYY’ group by col1
38. Summary
Result
● Presto was faster than Hive+Tez in all queries.
● Presto was faster than Hive+Tez in all data formats.
● With repetitive Querying, presto became faster.
● By Using RCfile, Presto was the most stable and fastest.
Next
● Benchmark from node scaling and data volumn
perspectives.
● Benchmark while using compression functions of
ORCfile.
● Benchmark with HDP2.2.
