This document describes building a REST job server for interactive Spark as a service using Livy. It discusses the history and challenges of running Spark jobs in Hue, introduces Livy as a Spark server, and details its local and YARN-cluster modes as well as session creation, execution flows, and interpreter support for Scala, Python, R and more. Magic commands are also covered for JSON, table, plotting and other output formats.
7. HISTORYāØ
V1: OOZIE
ā¢ It works
ā¢ Code snippet
THE GOOD
ā¢ Submit through Oozie
ā¢ Shell ac:on
ā¢ Very Slow
ā¢ Batch
THE BAD
workflow.xml
snippet.py
stdout
8. HISTORYāØ
V2: SPARK IGNITER
ā¢ It works beAer
THE GOOD
ā¢ Compiler Jar
ā¢ Batch only, no shell
ā¢ No Python, R
ā¢ Security
ā¢ Single point of failure
THE BAD Compile
Implement
Upload
json output
Batch
Scala
jar
Ooyala