Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
•Download as PPTX, PDF•
16 likes•8,279 views
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Similar to Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis (20)
More from DataWorks Summit/Hadoop Summit
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded
Recently uploaded (20)
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
- 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis Prabhjyot Singh & Jeff Zhang April, 14 2016
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more What’s Apache Zeppelin?
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Interactive Analysis 1.0 (Spark-shell)
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Interactive Analysis 2.0 (Zeppelin) Spark Interpreter
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Interactive Analysis 3.0 (Zeppelin + Livy) Livy Interpreter
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Open Source Activity
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Quick Stats: Zeppelin Incubated by Apache Foundation, first PR – Mar, 2015 Github history dates to Jul 2013, pre-incubation 9 Committers, 100+ contributors, growing list ~800 JIRAs filed ~900 PRs via the community Zeppelin just got a new friend “R” Zeppelin graduation in on the cards
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architecture & Usage
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Zeppelin Architecture Current Interpreter Support HDFS PySpark, SparkR, Spark Hive HBase Phoenix Shell SQL …
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Zeppelin Features Collate/Load Data Collate/Load data from existing data sources, load from external CSVs. i.e. Eureka, Smartsense Visualize Robust visualization mechanism to visualize data, and enable insights Collaborate Notebook base collaboration, export Notebooks, soon to be added, tagging to Notebook generated data
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Popular Usage Scenarios Customized Dashboards Intended for usage towards customized dashboards for Big Data clusters Security Analytics Understanding nature of data coming through multiple sources and analyzing the effects of it Bio-sciences Medical research companies are interested in using this for their research
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Bringing Multi-tenancy to Zeppelin
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Multi-Tenancy: Motivation Supporting workloads of multiple customers Supporting multiple LOBs (lines of business), on a single data systems Support fine grained audits Inability to provision capacity for multiple user groups Inability to Audit user actions, as all jobs are run via ‘zeppelin’ proxy user Inability to share state/data with other users as well Objectives Requirements
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Zeppelin Livy Interaction LDAP Zeppelin Shiro Spark Yarn Livy Ispark Group Interpreter SPNego: Kerberos Kerberos Security Across Zeppelin-Livy-Spark Livy APIs
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Deep dive on Livy
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What is Livy Livy ServerLivy Client Http Http (RPC) Http (RPC) Livy is an open source REST interface for interacting with Spark from anywhere. Spark Interactive Session SparkContext Spark Batch Session SparkContext
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why we need Livy with Zeppelin Reduce the pressure on client machine Make the job submission/monitoring easy Customize the job schedule
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Interactive Session – Create Session 2 1 3 4 curl -X POST --data '{"kind": "spark"}' -H "Content-Type: application/json" localhost:8998/sessions {"state":"starting","proxyUser":”null","id":1,"kind":"spark","log":[]} Request Response Livy Client Livy Server Spark Interactive Session SparkContext
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Interactive Session – Execute Code {"id":0,"state":"running","output":null} Request Response curl http://localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"sc.parallelize(0 to 100).sum()"}' 2 1 3 4 Livy Client Livy Server Spark Interactive Session SparkContext
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved SparkContext Sharing Livy Server Client 1 Client 2 Client 3 Session-1 Session-1 Session-2 Session-2 Session-1 SparkSession-1 SparkContext SparkSession-2 SparkContext
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Livy Security Client Livy Server (Impersonation) Shared SecretSpengo SparkSession • Only authorized users can launch spark session / submit code • Each user can access his own session • Only Livy server can submit job securely to spark session
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved SPNEGO Client (Kerbrose TGT) Livy Server (SPENGO enabled) Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO), often pronounced "spen-go” It is a GSSAPI "pseudo mechanism" used by client-server software to negotiate the choice of security technology. Http Get http://site/a.html Error 401 Unauthorized Http Get Request Authorization: Negotiation Http Get Request
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Impersonation Alice (Kerberos TGT) Shared Secret Bob (Kerberos TGT) Shared SecretSpengo Spengo Livy Server (super user: livy) Spark Session Spark Session
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Shared Secret 1. Livy Server generate secret key 2. Livy Server pass secret key to spark session when launching spark session 3. Use the secret key to communicate with each other Spark Session Shared Secret Livy Server
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Multi Tenant: Zeppelin Demo
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Q & A
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You
Editor's Notes
- So this is the interactive analysis 1.0 which is spark-shell. I think most of you are familiar with spark-shell if you have some experience of spark. Spark-shell provides a nice interactive environment for coding spark program. But it lacks lots of features for interactive analysis like visualization and code management.
- So this is why zeppelin comes. Zeppelin brings lots of nice features to interactive analysis, such as visualization, collaboration and etc. We can call it interactive analysis 2.0. By default Zeppelin use the native spark interpreter which has some limitation, such as not able to run spark in yarn-cluster mode. That means your driver runs on your client machine which may bring heavy pressure on your client machine. And besides that you can’t share spark context across multiple zeppelin instance.
- So now we leverage Livy as the spark interpreter of zeppelin. With Livy we can run spark interpreter in yarn cluster, and we can also share spark context across multiple zeppelin instance. We call zeppelin + livy as interactive analysis 3.0
- So what is livy ? Livy is an open source REST interface for interacting with Spark from anywhere. Here’s the diagram of the overall architecture of livy. There’s 3 layers, on the most left is the livy client and in the middle is livy server. Livy client commutate with livy server by using rest api. That means they communicate with Http protocol. Livy client can ask livy server to do lots of things, such as launching spark application, submit spark job,pull the job status and even submit one piece of spark code. There’s 2 kinds of spark session that livy support now. One is spark interactive session, another is spark batch session. Since today’s talk is about interactive analysis, so we will focus on the spark interactive session. In livy 0.1 the communication between livy server and spark session is Http, while in the latest code it is changed to RPC
- So overall livy is a central place to launching spark jobs. It brings the several benefits for us. First, It reduce the pressure on client machine. Nothing will run on the client machine except calling rest api Second, It makes the job submission/monitoring easy. Without livy you have to install spark on your client machine and use spark-submit to submit spark jobs. While in livy you just need to call the rest api The next is that you can customize the job schedule. Since all the job submission is through the livy-server, livy-server can do the job scheduling (This feature is not implemented yet, but it is possible)
- Now let’s talk about how livy works for the interactive session First we will talk about how livy create session. Before you submit any piece of code, you need to create session. Here we use the curl command to invoke the rest api. This is a POST request, and we specify the kind as spark, it can also be pyspark/sparkr, and we also need to specify the url of the rest api And this is the response we get. The response contains the state of the session, here it is starting, the proxyUser is null, Now let’s see how that request is routed. First livy client send request to livy server Then livy server will launch the session After the spark session session is created, it will send back its address to livy server, so that they can establish connection between livy server and spark session And finally livy server will send back the session status to livy client.
- Now let’s see how livy execute code Here’s the request we send, it contains the code that we want to execute and we also need to specify the rest api url. And here’s the response which contains the statement id, state, and output. Here we notice that the output is null, because this piece of code won’t finish in in short time, but we can get the output by calling another pull job status request. Now let’s see how this request is routed First livy client send request to livy server Livy server will forward the request to its spark session Spark session will execute the code and send back output to livy server Finally Livy server will send back output to livy client
- Now let’s talk about the SparkContext sharing Because clients don’t own the spark session, all the spark sessions are launched by livy server. So that makes the spark context sharing possible. Here we can see that client-1 and client-2 use the same spark session ( session-1). While client-3 use its own session (session-2) When the client interact with the livy server, he need to specify the session id, so as long as they specify the same session id, they are using the same spark context. Of course this is for non-secure mode, it is more complicated for secure mode.
- Now let’s talk about the security. Mainly there’s 3 secure problems we need to solve. First we need to make sure that only authorized users launch spark session. We don’t want everyone to launch spark session through livy server Second is that each user can access its own session. Third is only livy server can submit job securely to spark session To resolve these 3 problems we use several technics: spengo, impersonation and shared secret. I will talk about them one by one Spengo is used between livy client and livy server, it can make sure that only authroized users can launch spark session /submit code Impersonation is used to for make sure each user can access his own session. Without impersonation, all the spark session is launched as the user who launch the livy server process, but with impernation, the spark session is launched as the user in the client And the shared secret is used to protect the communication between livy server and spark session, only livy server and spark session know the shared secret
- First let’s talk about spengo. Spengo can make sure that only authorized user can launch spark session / submit code to livy server. The full name of spnego is Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) It is a GSSAPI "pseudo mechanism" used by client-server software to negotiate the choice of security technology. So it is pluggable with the underlying security technology, but most of often it is used with kerbrose. Now let see how that works. First the client will send the request to server Then the server will repponse with status code 401 which means unauthorized And then the client will send the request to server again, but this time it will put the kerborse service ticket information to the request Finally the server will authrozie the user with the ticket info and response with content of the page.
- The next thing is impersonation We want to protect each user’s session. We don’t want user Alice to access user bob’s session for security reason. The livy server process is launched by super user livy. Without impersonation all the spark session is launched as user livy, but with impersonation, the spark session can be launched as user of the client. This is very similar to the impersonation in hive server 2. So to enable this impersonation, we need to make the following configuration changes in core-site.xml
- The next thing we will talk about is the the share sceret. Once the spark session is started, it can accept request from outside, but we don’t want anyone to connect with the spark session except the livy server So here we use the shared scret to protect the communication between livy server and spark session. Only the livy server and spark session know the shared secret. Now let’s see how that works. Livy Server will generate secret key Livy Server pass secret key to spark session when launching spark Session Then they will use the secret key to communicate with each other