Hadoop World Spark Meetup: Interactive Spark in your Browser
•
5 likes•3,065 views
Hue now offers a Notebook application for interactively processing, visualizing and sharing data. Through a new Spark REST Job Server, Spark Python and Scala shells are available as well as Streaming. Those are ideal for doing some quick big data crunching from anywhere or a Web browser! This talk details the architecture of the REST API and Notebook UI as well as their integration with the Hadoop ecosystem. It also describes the alternatives we tried and the challenges that were faced. The capabilities will then be lived demo in Hue’s Notebook Application through a real life scenario combining Spark and Hadoop. Attendees will learn how to ramp-up on Spark and see how they could open it up to all their users and analyse even more data.
Recommended
Recommended
More Related Content
More from gethue
More from gethue (13)
Recently uploaded
Recently uploaded (20)
Hadoop World Spark Meetup: Interactive Spark in your Browser
- 1. INTERACTIVE SPARK IN YOUR BROWSER Spark Lightning Night -‐ Hadoop World September 2015
- 4. GOAL OF HUE WEB INTERFACE FOR ANALYZING DATA WITH APACHE HADOOP SIMPLIFY AND INTEGRATE FREE AND OPEN SOURCE —> “BIG NOTEBOOK” FOR HADOOP
- 8. • Married with full ecosystem • File, Job browsers • Create table wizards • Any language (Hive, Spark...) • Graphing • Export/Import/Sharing • MulT users • ImpersonaTon HADOOP WITH SPARK NOTEBOOK
- 10. • REST Web server in Scala • InteracTve Spark Sessions and Batch Jobs • Type IntrospecTon for VisualizaTon • Running sessions in YARN local • Backends: Scala, Python, R • Open Source: h[ps://github.com/cloudera/hue/ tree/master/apps/spark/java • Play with Curl h[p://gethue.com/how-‐to-‐use-‐the-‐ LIVY SPARK SERVER
- 11. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker Livy Server Scalatra Session Manager Session
- 12. LIVY WEB SERVER ARCHITECTURE Livy Server YARN Master Scalatra Spark Client Session Manager Session YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1
- 13. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 Livy Server Scalatra Session Manager Session
- 14. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 3 Livy Server Scalatra Session Manager Session
- 15. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 3 4 Livy Server Scalatra Session Manager Session
- 16. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 3 4 5 Livy Server Scalatra Session Manager Session
- 17. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 3 4 5 6 Livy Server Scalatra Session Manager Session
- 18. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 7 2 3 4 5 6 Livy Server Scalatra Session Manager Session
- 19. SESSION CREATION AND EXECUTION % curl -‐XPOST localhost:8998/sessions -‐d '{"kind": "spark"}' { "id": 0, "kind": "spark", "log": [...], "state": "idle" } % curl -‐XPOST localhost:8998/sessions/0/statements -‐d '{"code": " { "id": 0, "output": { "data": { "text/plain": "res0: Int = 2" }, "execution_count": 0, "status": "ok" }, "state": "available" }
- 20. DEMO TIME
- 21. • Stability and Scaling • Security • iPython/Jupyter interpreter and file format COMING SOON