Presented at ODSC West 2019: https://odsc.com/training/portfolio/big-fast-queries-with-presto-on-openshift-2/ Abstract: Next generation data platforms are embracing the proliferation of technologies that help organizations discover, catalog, process, and derive insight from their data. OpenShift, and OpenShift Container Storage are at the forefront of this transition and provide a foundation for building a self service environment for developers, data engineers, and data scientists. In this demo we'll share how Starburst Presto on OpenShift can power your interactive and ad-hoc data discovery. SQL on anything means fast, secure access to data in OpenShift Container Storage, and federated access to data anywhere. With Starburst on OpenShift you have access to the world’s fastest open source SQL query engine, enterprise ready, across clouds public and private. Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Comcast, GrubHub, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber. In the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.

1. With Presto on OpenShift
BIG FAST SQL
Kamil Bajda-Pawlikowski
Co-founder / CTO
Michael St-Jean
Principal Marketing Manager
1
ODSCWest-2019@SanFrancisco

2. CONFIDENTIAL Designator
PRIMER ON
PRESTO

3. 3
Presto: SQL-on-Anything
Deploy
Anywhere,
Query
Anything

4. Community
See more at our Wiki

5. Why Presto?
Community-driven
open source
project
High performance ANSI SQL
engine
• New Cost-Based Query Optimizer
• Proven scalability
• High concurrency
Separation of compute
and storage
• Scale storage and
compute independently
• No ETL or data integration
necessary to get to
insights
• SQL-on-anything
No vendor lock-in
• No Hadoop distro vendor lock-in
• No storage engine vendor lock-in
• No cloud vendor lock-in

6. Enterprise edition
© 2019 6
Founded by Presto committers:
● Over 4 years of contributions to Presto
● Presto distro for on-prem and cloud env
● Supporting large customers in production
● Enterprise subscription add-ons (ODBC,
Ranger, Sentry, Oracle, Teradata, K8S)
Notable features contributed:
● ANSI SQL syntax enhancements
● Execution engine improvements
● Security integrations
● Spill to disk
● Cost-Based Optimizer
https://www.starburstdata.com/presto-enterprise/

7. CBO off
CBO on
https://www.starburstdata.com/presto-benchmarks/
Benchmark results

8. © 2019

9. Administrative challenges
● Configuring and managing clusters
● Autotuning properties based on the hardware provisioned
● High Availability for Presto Coordinator
● Scaling cluster elastically based on query load
● Gracefully decommissioning Presto Workers to avoid killing queries
● Monitoring of hardware and software layers
https://www.starburstdata.com/technical-blog/presto-on-kubernetes/

10. https://docs.starburstdata.com/latest/kubernetes.html
Presto on OpenShift
Presto Worker
Pod
Presto Worker
Pod
10
Presto Coordinator
Pod
Presto Worker
Pod
Horizontal Pod
Autoscaler (HPA)
Presto Operator
K8s Operator
Presto
Service
Hive Metastore
Service
Pod
Hadoop / Hive
RDBMS

11. Now available
in OpenShift
Catalog!

12. CONFIDENTIAL Designator
CEPH OBJECT
INTRODUCTION

13. ● Massively scalable
○ 10’s-100’s of PBs
○ Billions of objects
○ 100’s of gigabits
● Erasure coding drives storage efficiency
● High level of fidelity with S3 API
● Open source software - LGPL 2.1 / 3

14. ● Operators: OCS, Rook-Ceph, Rook-Noobaa
● Ceph for block (RWO), file (RWO/RWX), object (S3)
● Noobaa for data federation in the hybrid cloud

15. Hive connector configuration example (hive.properties)
connector.name=hive-hadoop2
hive.metastore.uri=thrift://metastore.example.com:9083
hive.s3.endpoint=s3.example.com
hive.s3.aws-access-key=ACCESS_KEY
hive.s3.aws-secret-key=SECRET_KEY
hive.s3.use-instance-credentials=false
hive.s3.staging-directory=/tmp
hive.s3.ssl.enabled=true

16. presto> CREATE SCHEMA hive.s3_export WITH (location =
's3://my_bucket/some/path');
presto> CREATE TABLE hive.s3_export.my_table
WITH (format = 'ORC')
AS <source query>;

17. orders
customer
JOIN Query
Federated JOIN Query
Data Scientist
tpc-hschema

18. Thank You!
18
Twitter: @starburstdata @prestosql
Blog: www.starburstdata.com/technical-blog/
Newsletter: www.starburstdata.com/newsletter
© 2019