Presto SQL at Wayfair meetup in Boston https://www.meetup.com/DataCouncil-AI-Boston-Data-Engineering-Science/events/262818999/ Abstract: Presto has become the essential tool for Data Scientists and Analysts at Wayfair. Presto is relatively new at Wayfair. It was implemented about a year ago and has been actively used in the past 6 months. Attend this session to understand why Wayfair decided to implement presto, how we architected our cluster, configuration choices we made, some common issues we faced and where we are heading towards next. By breaking down our problems and approach with Presto, it will help describe some of the challenges we face, and provide color to the decisions we’ve made. Speaker Bio: Krishna Ravishankar is a DevOps Engineer working on the BigData and Messaging Platforms at Wayfair. He joined the team about 2 years ago with a little knowledge of how distributed systems work, now Krishna is one of the leads for some of the platforms (Presto, Kafka) the team own. Krishna was invloved in bringing Presto into Wayfair, he designed and implemented a production presto cluster which is now used by hundereds of users running ad-hoc queries. He’s currently now involved in building a new read/write cluster using Starburst distro.

1. Presto at Wayfair
Krishna Ravishankar

2. 2
• Problem Statement
• Why use Presto ?
• Presto at Wayfair
• Presto Clients
• Presto adoption
• Moving towards
• Monitoring
• Q/A
Agenda

3. 3
Problem Statement

4. 4
1. Optimize Hive queries
1. Setting up queues to prioritize batch jobs
1. Throttle users to 2 ad-hoc queries
1. Move jobs from Hive to Spark
1. Conduct SME training session for both Hive and
Spark
Remedies

5. 5
Why Presto ?
● It’s VERY fast!
● It saves hadoop resources: by using presto, you enable more development work to
be done as other teams test their pyspark pipelines on the cluster
● Unlike spark which requires more expertise and set up, presto is quick to set up.
● You can combine data sources in different places (SQL and hive data in one place)

6. 6
Presto at Wayfair

7. 7
Presto ad-hoc (Read
Only Cluster)
301 VM’s (8*64)
with 1
Coordinator, 300
Workers
Total available
Memory 20TB
Total CPU
available 2800
vcores
Presto CLI
Presto at Wayfair

8. 8 Adoption - before
140K Queries
80K Queries
40%

9. 9 Adoption - after

10. 10 Query Throttling
● SELECT only
● 2 queries per user
● 2 queued queries
per user
● Increased the time
limit from 5 to 10
mins
avg
execution
time - 51 sec

11. 11 Moving towards Starburst

12. 12 OSS Presto Vs Starburst
Starburst
Presto
open
source
Note:
1. CBO turned
manually on
OSS presto
2. Starburst has
CBO turned on
by default
3. CBO improved
query
performance by
3-10X

13. 13 Monitoring Presto

14. 14
1. Migrating jobs
1. Upgrading our existing presto cluster to use Starburst distribution
1. BigQuery Vs Presto
What’s Next?

15. 15
THANK YOU
Questions?
Krishna Ravishankar
DevOps Engineer
kravishankar@wayfair.com
https://www.linkedin.com/in/ravishankarkrishnakumar/