Spark Compute as a Service at Paypal with Prabhu Kasinathan
•
4 likes•2,181 views
Apache Spark is a gift to the big data community, which adds tons of new features on every release. However, it’s difficult to manage petabyte-scale Hadoop clusters with hundreds of edge nodes, multiple Spark releases and demonstrate operational efficiencies and standardization. In order to address these challenges, Paypal has developed and deployed a REST0based Spark platform: Spark Compute as a Service (SCaaS),which provides improved application development, execution, logging, security, workload management and tuning. This session will walk through the top challenges faced by PayPal administrators, developers and operations and describe how Paypal’s SCaaS platform overcomes them by leveraging open source tools and technologies, like Livy, Jupyter, SparkMagic, Zeppelin, SQL Tools, Kafka and Elastic. You’ll also hear about the improvements PayPal has added, which enable it to run greater than 10,000 Spark applications in production effectively.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to Spark Compute as a Service at Paypal with Prabhu Kasinathan
Similar to Spark Compute as a Service at Paypal with Prabhu Kasinathan (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Spark Compute as a Service at Paypal with Prabhu Kasinathan
- 1. Prabhu Paypal, Inc. Spark Compute as a Service @ Paypal
- 2. Scale
- 3. Paypal Scale Business $354B Total Payment up 28% YoY $102B Mobile Payment up 55% YoY 6.1B Total Transactions up 24% YoY 2.0B Mobile Transactions up 43% YoY • One of the world’s largest internet payment companies • 203+M active accounts on 200 markets around the world • PayPal platform includes Braintree, Venmo, Paydiant, PP Credit and Xoom
- 4. Paypal Scale Core Data Platform 70+PB Data 40,000+ Yarn Jobs Per Day 15+ Hadoop Clusters 5+ Compute
- 7. Challenges
- 8. Challenges • Need extensive support and maintenance for CLI • Need to deploy entire stack of software • Need to sync configurations across systems • Need extensive testing of jobs before any upgrade Batch Edge Nodes Interactive Edge Nodes Job Administrators
- 9. Challenges Batch Edge Nodes Interactive Edge Nodes Job Developers • No REST-friendly • No low-latency/sub-seconds execution • No cache sharing across jobs • No modularity and easy-restartability
- 10. Challenges Batch Edge Nodes Interactive Edge Nodes Job Analysts/Scientists • No easy way of interactive applications • No multi-tenancy support and private workspace • No direct spark sql execution • No Kerberos integration
- 11. Challenges • Different ways of jobs execution and coding standards • No uniform logging, monitoring and alerting • Limited audit and control • No statement level history or metrics Batch Edge Nodes Interactive Edge Nodes Job Operations/Security
- 12. Building SCaaS Spark Compute Platform
- 14. Batch Interactive JobJob Server Building SCaaS Adding REST Job Server
- 15. Batch Interactive Job LIVY GRID Job Server Building SCaaS Adding HAand Enhance Livy PayPal Livy Version ü Multi-Nodes High Availability ü Kerberos Authentication Changes ü SQL Interpreter ü Session Manager Enhancements ü Session GC Improvements ü Plug-in Logger ü Yarn Poll Re-architecture ü Multiple Spark Versions Support ü White/Black listUser Authentication ü Dockers ü Hbase Support ü Flink/Beam Support
- 16. Batch Interactive Job LIVY GRID Job Server Livy API NAS Batch Building SCaaS Adding Livy API and Utilities Batch Utilities ü startSparkBatch ü stopSparkBatch ü listSparkBatch ü startSparklingWater ü stopSparklingWater ü startSparkSql ü stopSparkSql ü startSparkSession ü execSparkFile ü execSparkCode ü stopSparkSession ü listSparkSession ü livy-spark.jar Interactive Utilities ü livy-spark
- 17. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive Building SCaaS Adding Zeppelin
- 18. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive In InIn In Building SCaaS Adding Jupyter
- 19. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive In InIn In Interactive Interactive Building SCaaS Adding SQL Client
- 20. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive In InIn In Interactive Interactive Sparkling Water Interactive Interactive Building SCaaS Adding Sparkling Water
- 21. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive In InIn In Interactive Interactive Sparkling Water Interactive Log Building SCaaS Adding Spark Logger Interactive
- 22. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive In InIn In Interactive Interactive Sparkling Water Interactive Log Log Building SCaaS Adding Livy Logger Interactive
- 23. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive In InIn In Interactive Interactive Sparkling Water Interactive Interactive Log Log Indexing Search Building SCaaS Adding Search Engine
- 24. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive In InIn In Interactive Interactive Sparkling Water Interactive Interactive Log Log Indexing Search Metrics Building SCaaS Adding Monitoring/Alerting
- 25. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive In InIn In Interactive Interactive Sparkling Water Interactive Interactive Log Log Indexing Search Metrics History Server Building SCaaS Adding Standard Tools
- 26. Batch Job LIVY GRID Job Server Livy API NAS Batch Interactive Interactive In InIn In Interactive Interactive Sparkling Water Interactive Interactive Log Log Indexing Search Metrics History Server Building SCaaS Here, you go! The SCaaS Framework
- 27. SCaaS Benefits
- 28. Administrators ü Less maintenance on CLI ü Deploy software stack only on Job Server ü Configurations at one place ü Easy platform/softwareupgrade Developers ü REST-friendly and Docker-friendly ü Low-latency/sub-seconds execution ü Sharing cache across jobs ü Modularity and easy restartability Analysts/Scientists ü User friendly interactive applications ü Multi-tenancy and Private workspace ü Direct spark sql execution ü Kerberos Support Operations/Security ü Standardized coding andunified execution ü Uniformed logging, monitoring and alerting ü Fine-grained audit ü Complete statement level history and metrics SCaaS Benefits
- 29. SCaaS Demo