Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Problematic Queries
•
0 likes•143 views
John submits a query and expects it to run smoothly. Based on his prior experience, he anticipates the query to finish in 20 mins. Scenario-1: John’s query finishes execution in the expected timeframe and doesn’t impact any other concurrent query in the workload. Scenario-2: John’s query takes twice the expected time, and also slows down multiple other concurrent queries. John now wonders “should I have submitted this query?”.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Problematic Queries
Similar to Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Problematic Queries (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Problematic Queries
- 1. Query or Not to Query Ask Unravel Prajakta Kalmegh, Principal Engineer Yusaku Sako, Head of Data Science
- 2. Data Science@ Prajakta Kalmegh ▪ pkalmegh@unraveldata.com ▪ https://www.linkedin.com/in/pkalmegh/ Yusaku Sako ▪ ysako@unraveldata.com ▪ https://www.linkedin.com/in/yusaku-sako/
- 3. Experienced Team Strong Market Validation A Microsoft M12 Company Broad Technology and Cloud Platform Coverage Our Pedigree
- 5. Select right tech for the app, infrastructure, environment and cluster Debug code and pipelines, predict issues and assist in optimizing apps Check for app correctness and resource efficiency Container sizing, Scheduling and Cluster selection Tuning apps and eliminating rogue apps Proactive and automated actions to maintain SLAs CONTINUOUS INTEGRATION C O N TIN U O U S O PTIM IZA TIO N
- 9. Meet John 8:00 am 11:00 am 11:30 am 4:30 pm Still not working?
- 10. Why is this hard? What if I tweak my query? Did my resource requirements change? Is my data skewed on xxx? Is the cluster bottlenecked?
- 11. Unravel brings data-driven insights as you code
- 12. The notebook Demo 2Q | | !2Q
- 13. What Unravel exploits? ▪ Users often issue similar queries ▪ Same challenges faced ▪ Same mistakes repeated ▪ By the same user, by other users Holistic view of what worked and what did not
- 15. When is a good time to schedule? These are my resource requirements, should I schedule now or later? This report needs to be ready before Monday morning, my start time is flexible I have a new workload to schedule, what is a good time to start it?
- 16. Finding the missing piece Is the cluster slow (er) ? Has the query needs changed over time? Is it both?
- 17. Unravel uses predictive analytics to time it right
- 18. The timeit Demo while (!best) { # find better }
- 19. What Unravel exploits? ▪ Cluster utilization timeseries ▪ Query execution variability data ▪ Similar resource profiles and consumption history Holistic view of when it worked and when it didn’t
- 21. DELAYED Delayed: Increase in Input Data Size Detected
- 22. Delayed: Increase in Input Data Size Detected DELAYED
- 23. Architecture UNRAVELDAEMONS UnravelAPI Query Historical Executions Cluster State Indicators Query Quality Predictor Best Slot Finder AskUnravel
- 24. Already Scheduled Scheduled Query User Adhoc Query New Schedule Predict query issues Predict cluster issues Predict delays Submit Query ✓ SLA-bound ✓ Responsive wantstosubmit/track askunravel Submit Query Better Slots are xxx On Track Delayed by xxx Use-Cases Hold and submit lateraskunravel Better Slots are xxx Operational Insights Optimize Deployment Speedup Development
- 25. Recap: Detect Issues as early as possible 8:00 am 8:30 am Enjoy your day!
- 26. Thank you for watching! Signup for a free trial today https://bit.ly/3mo2ira
- 27. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.