Spark or Hadoop: Is it an either-or proposition

Spark or Hadoop: Is it an either-or
proposition?
By Slim Baltagi (@SlimBaltagi)
Big Data Practice Director
Advanced Analytics LLC
OR
XOR ??
Los Angeles Spark Users Group
March 12, 2015

Your Presenter – Slim Baltagi
2
• Sr. Big Data Solutions Architect
living in Chicago.
• Over 17 years of IT and business
experiences.
• Over 4 years of Big Data
experience working on over 12
Hadoop projects.
• Speaker at Big Data events.
• Creator and maintainer of the
Apache Spark Knowledge
Base:
http://www.SparkBigData.com
with over 4,000 categorized
Apache Spark web resources.
@SlimBaltagi
https://www.linkedin.com/in/slimbalta
gi
sbaltagi@gmail.com
Disclaimer: This is a vendor-independent talk that expresses my own
opinions. I am not endorsing nor promoting any product or vendor mentioned in
this talk.

Agenda
I. Motivation
II. Big Data, Typical Big Data
Stack, Apache Hadoop,
Apache Spark
III. Spark with Hadoop
IV. Spark without Hadoop
V. More Q&A
3

I. Motivation
1. News
2. Surveys
3. Vendors
4. Analysts
5. Key Takeaways
4

1. News
• Is it Spark 'vs' OR 'and' Hadoop?
• Apache Spark: Hadoop friend or foe?
• Apache Spark: killer or savior of Apache Hadoop?
• Apache Spark's Marriage To Hadoop Will Be Bigger
Than Kim And Kanye.
• Adios Hadoop, Hola Spark!
• Apache Spark: Moving on from Hadoop
• Apache Spark Continues to Spread Beyond
Hadoop.
• Escape From Hadoop!
• Spark promises to end up Hadoop, but in a good
way
5

2. Surveys
• "Hadoop's historic focus on batch processing of data
was well supported by MapReduce, but there is an
appetite for more flexible developer tools to support
the larger market of 'mid-size' datasets and use cases
that call for real-time processing.” 2015 Apache Spark
Survey by Typesafe. January 27, 2015.
http://www.marketwired.com/press-release/survey-indicates-apache-spark-
gaining-developer-adoption-as-big-datas-projects-1986162.htm
• Apache Spark: Preparing for the Next Wave of
Reactive Big Data, January 27, 2015 by Typesafe
http://typesafe.com/blog/apache-spark-preparing-for-the-next-wave-of-reactive-
big-data
6

Apache Spark Survey 2015 by
Typesafe - Quick Snapshot
7

3. Vendors
8
• Spark and Hadoop: Working Together. January 21,
2014 by Ion Stoica https://databricks.com/blog/2014/01/21/spark-and-
hadoop.html
• Uniform API for diverse workloads over diverse
storage systems and runtimes.
Source: Slide 16 of ‘Spark's Role in the Big Data Ecosystem (Spark
Summit 2014). November 2014. Matei
Zahariahttp://www.slideshare.net/databricks/spark-summit2014
• "The goal of Apache Spark is to have one engine for all
data sources, workloads and environments.”
Source: Slide 15 of ‘New Directions for Apache Spark in 2015,
February 20, 2015. Strata + Hadoop Summit. Matei Zaharia
http://www.slideshare.net/databricks/new-directions-for-apache-spark-in-2015

3. Vendors
• “Spark is already an excellent piece of software and is
advancing very quickly. No vendor — no new project —
is likely to catch up. Chasing Spark would be a waste
of time, and would delay availability of real-time analytic
and processing services for no good reason. ”
Source: MapReduce and Spark, December, 30,2013
http://vision.cloudera.com/mapreduce-spark/
• “Apache Spark is an open source, parallel data
processing framework that complements Apache
Hadoop to make it easy to develop fast, unified Big Data
applications combining batch, streaming, and interactive
analytics on all your data.”
http://www.cloudera.com/content/cloudera/en/products-and-
services/cdh/spark.html
9

3. Vendors
• “Apache Spark is a general-purpose engine for large-
scale data processing. Spark supports rapid application
development for big data and allows for code reuse
across batch, interactive and streaming applications.
Spark also provides advanced execution graphs with in-
memory pipelining to speed up end-to-end application
performance.” https://www.mapr.com/products/apache-spark
• MapR Adds Complete Apache Spark Stack to its
Distribution for Hadoop
https://www.mapr.com/company/press-releases/mapr-adds-complete-apache-
spark-stack-its-distribution-hadoop
10

3. Vendors
• “Apache Spark provides an elegant, attractive
development API and allows data workers to rapidly
iterate over data via machine learning and other
data science techniques that require fast, in-
memory data processing.”
http://hortonworks.com/hadoop/spark/
• Hortonworks: A shared vision for Apache Spark on
Hadoop. October 21,
2014https://databricks.com/blog/2014/10/31/hortonworks-a-shared-vision-for-
apache-spark-on-hadoop.html
• “At Hortonworks, we love Spark and want to help our
customers leverage all its benefits.” October 30th, 2014
http://hortonworks.com/blog/improving-spark-data-pipelines-native-yarn-
integration/
11

4. Analysts
• Is Apache Spark replacing Hadoop or complementing
existing Hadoop practice?
• Both are already happening:
• With uncertainty about “what is Hadoop” there is no
reason to think solution stacks built on Spark, not
positioned as Hadoop, will not continue to proliferate
as the technology matures.
• At the same time, Hadoop distributions are all
embracing Spark and including it in their offerings.
Source: Hadoop Questions from Recent Webinar Span Spectrum.
February 25, 2015.http://blogs.gartner.com/merv-adrian/2015/02/25/hadoop-
questions-from-recent-webinar-span-spectrum/
12

4. Analysts
• “After hearing the confusion between Spark and
Hadoop one too many times, I was inspired to write a
report, The Hadoop Ecosystem Overview, Q4 2104.
• For those that have day jobs that don’t include constantly
tracking Hadoop evolution, I dove in and worked with
Hadoop vendors and trusted consultants to create a
framework.
• We divided the complex Hadoop ecosystem into a core
set of tools that all work closely with data stored in
Hadoop File System and extended group of
components that leverage but do not require it.”
Source: Elephants, Pigs, Rhinos and Giraphs; Oh My! – It's Time To
Get A Handle On Hadoop. Posted by Brian Hopkins on November 26,
2014
http://blogs.forrester.com/brian_hopkins/14-11-26-
elephants_pigs_rhinos_and_giraphs_oh_my_its_time_to_get_a_handle_on_hadoop
13

5. Key Takeaways
1. News: Big Data is no longer a Hadoop
monopoly!
2. Surveys: Listen to what Spark developers are
saying!
3. Vendors: <Hadoop Vendor>-tinted goggles!?
FUD is still being ‘offered’ by some Hadoop
vendors. Claims need to be contextualized.
4. Analysts: Thorough understanding of the
market dynamics !?
14

II. Big Data, Typical Big Data
Stack, Hadoop, Spark,
1. Big Data
2. Typical Big Data Stack
3. Apache Hadoop
4. Apache Spark
5. Key Takeaways
15

1. Big Data
• Big Data is still one of the most inflated buzzword of
the last years.
• Big Data is a broad term for data sets so large or
complex that traditional data processing tools are
inadequate. http://en.wikipedia.org/wiki/Big_data
• Hadoop is becoming a traditional tool. Above
definition is inadequate!?
• “Big Data refers to datasets and flows large enough
that has outpaced our capability to store, process,
analyze, and understand.” Amir H. Payberah,
Swedish Institute of Computer Science (SICS).
16

3. Apache Hadoop
• Apache Hadoop as an example of a Typical Big Data
Stack.
• Hadoop ecosystem = Hadoop Stack + many other tools
(either open source and free or commercial ones).
• Big Data Ecosystem Dataset http://bigdata.andreamostosi.name/
Incomplete but a useful list of Big Data related projects
packed into a JSON dataset.
• "Hadoop's Impact on Data Management's Future" - Amr
Awadallah (Strata + Hadoop 2015). February 19, 2015: Watch
video at 2:36 on ‘Hadoop Isn’t Just Hadoop Anymore’ for a picture
representing the evolution of Apache Hadoop.
https://www.youtube.com/watch?v=1KvTZZAkHy0
18

4. Apache Spark
• Apache Spark as an example of a Typical Big Data Stack.
• Apache Spark provides you Big Data computing and more:
• BYOS: Bring Your Own Storage.
• BYOC: Bring Your Own Cluster.
• Spark Core: http://sparkbigdata.com/component/tags/tag/11-core-spark
• Spark Streaming: http://sparkbigdata.com/component/tags/tag/3-spark-
streaming
• Spark SQL: http://sparkbigdata.com/component/tags/tag/4-spark-sql
• MLlib (Machine Learning) http://sparkbigdata.com/component/tags/tag/5-
mllib
• GraphX: http://sparkbigdata.com/component/tags/tag/6-graphx
• Spark ecosystem is emerging fast with roots from BDAS:
Berkley Data Analytics Stack and new tools from both the open
source community and commercial one. I’m compiling a list.
Stay tuned!
19

5. Key Takeaways
1. Big Data: Still one of the most inflated
buzzword!?
2. Typical Big Data Stack: Big Data Stacks look
similar on paper. Aren’t they!?
3. Apache Hadoop: Hadoop is no longer
‘synonymous’ of Big Data!
4. Apache Spark: Emergence of the Apache
Spark ecosystem.
20

1. Evolution
2. Transition
3. Integration
4. Complementarity
5. Key Takeaways
21

1. Evolution of Programming APIs
• MapReduce in Java is like assembly code of Big
Data! http://wiki.apache.org/hadoop/WordCount
• Pig http://pig.apache.org
• Hive http://hive.apache.org
• Scoobi: A Scala productivity framework for Hadoop
https://github.com/NICTA/scoobi
• Cascading http://www.cascading.org/
• Scalding: A Scala API for Cascading http://twitter.com/scalding
• Crunch http://crunch.apache.org
• Scrunch http://crunch.apache.org/scrunch.html
22

1. Evolution of Compute Models
When the Apache Hadoop project started in 2007,
MapReduce v1 was the only choice as a compute model
(Execution Engine) on Hadoop. Now we have, in addition
to MapReduce v2, Tez, Spark and Flink:
23
• Batch • Batch
• Interactive
• Batch
• Interactive
• Near-Real
time
• Batch
• Interactive
• Real-Time
• Iterative
• 1st
Generation
• 2nd
Generation
• 3rd
Generation
• 4th
Generation

1. Evolution:
• This is how Hadoop MapReduce is branding itself: “A
YARN-based system for parallel processing of large data
sets. http://hadoop.apache.org
• Batch, Scalability, Abstractions ( See slide on evolution of
Programming APIs), User Defined Functions (UDFs)…
• Hadoop MapReduce (MR) works pretty well if you can
express your problem as a single MR job. In practice,
most problems don't fit neatly into a single MR job.
• Need to integrate many disparate tools for advanced
Big Data Analytics for Queries, Streaming Analytics,
Machine Learning and Graph Analytics.
24

1. Evolution:
• Tez: Hindi for “speed”
• This is how Apache Tez is branding itself: “The
Apache Tez project is aimed at building an
application framework which allows for a complex
directed-acyclic-graph of tasks for processing
data. It is currently built atop YARN.”
Source: http://tez.apache.org/
• Apache™ Tez is an extensible framework for
building high performance batch and
interactive data processing applications,
coordinated by YARN in Apache Hadoop.
25

1. Evolution:
• ‘Spark’ for lightning fast speed.
• This is how Apache Spark is branding itself:
“Apache Spark™ is a fast and general engine for
large-scale data processing.” https://spark.apache.org
• Apache Spark is a general purpose cluster
computing framework, its execution model
supports wide variety of use cases: batch,
interactive, near-real time.
• The rapid in-memory processing of resilient
distributed datasets (RDDs) is the “core
capability” of Apache Spark.
26

1. Evolution: Apache Flink
• Flink: German for “nimble, swift, speedy”
• This is how Apache Flink is branding itself: “Fast and
reliable large-scale data processing engine”
• Apache Flink http://flink.apache.org/ offers:
• Batch and Streaming in the same system
• Beyond DAGs (Cyclic operator graphs)
• Powerful, expressive APIs
• Inside-the-system iterations
• Full Hadoop compatibility
• Automatic, language independent optimizer
• ‘Flink’ Tag at SparkBigData.com
http://sparkbigdata.com/component/tags/tag/27-flink
27

Hadoop MapReduce vs. Tez vs. Spark
Criteria
License Open Source
Apache 2.0, version
2.x
Open Source,
Apache 2.0,
version 0.x
Open Source,
Apache 2.0, version
1.x
Processing
Model
On-Disk (Disk-
based
parallelization),
Batch
On-Disk, Batch,
Interactive
In-Memory, On-Disk,
Batch, Interactive,
Streaming (Near Real-
Time)
Language written
in
Java Java Scala
API [Java, Python,
Scala], User-Facing
Java,[
ISV/Engine/Tool
builder]
[Scala, Java, Python],
User-Facing
Libraries None, separate tools None [Spark Core, Spark
Streaming, Spark SQL,
MLlib, GraphX]
28

Criteria
Installation Bound to Hadoop Bound to Hadoop Isn’t bound to
Hadoop
Ease of Use Difficult to program,
needs abstractions
No Interactive mode
except Hive, Pig
Difficult to program
No Interactive
mode except Hive,
Pig
Easy to program,
no need of
abstractions
Interactive mode
Compatibilit
y
to data types and data
sources is same
to data types and
data sources is
same
to data types and
data sources is
same
YARN
integration
YARN application Ground up YARN
application
Spark is moving
towards YARN
29

Criteria
Deployment YARN YARN [Standalone, YARN*,
SIMR, Mesos, …]
Performance - Good performance
when data fits into
memory
- performance
degradation otherwise
Security More features and
projects
More
features and
projects
Still in its infancy
30
* Partial support

IV. Spark with Hadoop
1. Evolution
2. Transition
3. Integration
4. Complementarity
5. Key Takeaways
31

2. Transition
• Existing Hadoop MapReduce projects can
migrate to Spark and leverage Spark Core as
execution engine:
1. You can often reuse your mapper and
reducer functions and just call them in
Spark, from Java or Scala.
2. You can translate your code from
MapReduce to Apache Spark. How-to:
Translate from MapReduce to Apache Spark
http://blog.cloudera.com/blog/2014/09/how-to-translate-from-mapreduce-to-
apache-spark/
32

2. Transition
3. The following tools originally based on Hadoop
MapReduce are being ported to Apache Spark:
• Pig, Hive, Sqoop, Cascading, Crunch, Mahout, …
33

 Pig on Spark (Spork)
• Run Pig with “–x spark” option for an easy migration
without development effort.
• Speed up your existing pig scripts on Spark ( Query,
Logical Plan, Physical Pan)
• Leverage new Spark specific operators in Pig such as
Cache
• Still leverage many existing Pig UDF libraries
• Pig on Spark Umbrella Jira (Status: Passed end-to-end test
cases on Pig, still Open) https://issues.apache.org/jira/browse/PIG-4059
• Fix outstanding issues and address additional Spark functionality
through the community
• ‘Pig on Spark’ Tag at SparkBigData.com
http://sparkbigdata.com/component/tags/tag/19
34

 Hive on Spark (Currently in Beta,
Expected in Hive 1.1.0)
• New alternative to using MapReduce or Tez:
hive> set hive.execution.engine=spark;
• Help existing Hive applications running on
MapReduce or Tez easily migrate to Spark without
development effort.
• Exposes Spark users to a viable, feature-rich de facto
standard SQL tool on Hadoop.
• Performance benefits especially for Hive queries,
involving multiple reducer stages.
• Hive on Spark Umbrella Jira (Status: Open). Q1 2015
https://issues.apache.org/jira/browse/HIVE-7292
35

Hive on Spark (Currently in Beta,
Expected in Hive 1.1.0)
• Design
http://blog.cloudera.com/blog/2014/07/apache-hive-on-apache-spark-
motivations-and-design-principles/
• Getting Started
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark:+Getting+Start
ed
• Hive on Spark, February 11, 2015, Szehon Ho,
Clouderahttp://www.slideshare.net/trihug/trihug-feb-hive-on-spark
• Hive on spark is blazing fast... or is it? Carter Shanklin and
Mostapah Mokhtar (Hortonworks). February 20, 2015.
http://www.slideshare.net/hortonworks/hive-on-spark-is-blazing-fast-or-is-it-final
• ‘Hive on Spark’ Tag at SparkBigData.com
36

 Sqoop on Spark
(Expected in Sqoop 2)
• Sqoop ( a.k.a from SQL to Hadoop) was initially
developed as a tool to transfer data from RDBMS to
Hadoop.
• The next version of Sqoop, referred to as Sqoop2
supports data transfer across any two data sources.
• Sqoop 2 Proposal is still under
discussion.https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Pro
posal
• Sqoop2: Support Sqoop on Spark Execution Engine (Jira
Status: Work In Progress). The goal of this ticket is to support a
pluggable way to select the execution engine on which we can run
the Sqoop jobs. https://issues.apache.org/jira/browse/SQOOP-1532
37

(Expected in 3.1 release)
• Cascading http://www.cascading.org is an application
development platform for building data applications on
Hadoop.
• Support for Apache Spark is on the roadmap and will be
available in Cascading 3.1 release.
Source: http://www.cascading.org/new-fabric-support/
• Spark-scalding is a library that aims to make the
transition from Cascading/Scalding to Spark a little
easier by adding support for Cascading Taps, Scalding
Sources and the Scalding Fields API in Spark. Source:
http://scalding.io/2014/10/running-scalding-on-apache-spark/
38

Apache Crunch
• The Apache Crunch Java library provides a
framework for writing, testing, and running
MapReduce pipelines. https://crunch.apache.org
• Apache Crunch 0.11 releases with a
SparkPipeline class, making it easy to migrate
data processing applications from MapReduce
to Spark.
https://crunch.apache.org/apidocs/0.11.0/org/apache/crunch/impl/spark/Spark
Pipeline.html
• Running Crunch with Spark
http://www.cloudera.com/content/cloudera/en/documentation/core/v5-2-
x/topics/cdh_ig_running_crunch_with_spark.html
39

(Expec (Expected in Mahout 1.0 )
• Mahout News: 25 April 2014 - Goodbye MapReduce:
Apache Mahout, the original Machine Learning (ML)
library for Hadoop since 2009, is rejecting new
MapReduce algorithm
implementations.http://mahout.apache.org
• Integration of Mahout and Spark:
• Reboot with new Mahout Scala DSL for Distributed
Machine Learning on Spark: Programs written in this
DSL are automatically optimized and executed in
parallel on Apache Spark.
• Mahout Interactive Shell: Interactive REPL shell for
Spark optimized Mahout DSL.
http://mahout.apache.org/users/recommender/intro-cooccurrence-spark.html
40

(Expected in Mahout 1.0 )
• Playing with Mahout's Spark Shell
https://mahout.apache.org/users/sparkbindings/play-with-shell.html
• Mahout scala and spark bindings. Dmitriy Lyubimov,
April 2014
http://www.slideshare.net/DmitriyLyubimov/mahout-scala-and-spark-bindings
• Co-occurrence Based Recommendations with
Mahout, Scala and Spark. Published on May 30, 2014
http://www.slideshare.net/sscdotopen/cooccurrence-based-recommendations-
with-mahout-scala-and-spark
• Mahout 1.0 Features by Engine (unreleased)-
MapReduce, Spark, H2O, Flink
http://mahout.apache.org/users/basics/algorithms.html
41

1. Evolution
2. Transition
3. Integration
4. Complementarity
5. Key Takeaways
42

3. Integration
Service Open Source Tool
Storage/Servi
ng Layer
Data Formats
Data
Ingestion
Services
Resource
Management
Search
SQL
43

3. Integration:
• Spark was designed to read and write data from and to
HDFS, as well as other storage systems supported by
Hadoop API, such as your local file system, Hive, HBase,
Cassandra and Amazon’s S3.
• Stronger integration between Spark and HDFS caching
(SPARK-1767) to allow multiple tenants and processing
frameworks to share the same in-memory
https://issues.apache.org/jira/browse/SPARK-1767
• Use DDM: Discardable Distributed Memory
http://hortonworks.com/blog/ddm/ to store RDDs in memory.This
allows many Spark applications to share RDDs since they
are now resident outside the address space of the
application. Related HDFS-5851 is planned for Hadoop
3.0 https://issues.apache.org/jira/browse/HDFS-5851
44

3. Integration:
• Out of the box, Spark can interface with HBase as it has
full support for Hadoop InputFormats via
newAPIHadoopRDD. Example: HBaseTest.scala from
Spark Code.
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apach
e/spark/examples/HBaseTest.scala
• There are also Spark RDD implementations available
for reading from and writing to HBase without the need
of using Hadoop API anymore: Spark-HBase Connector
https://github.com/nerdammer/spark-hbase-connector
• SparkOnHBase is a project for HBase integration with
Spark. Status: Still in experimentation and no timetable for
possible support. http://blog.cloudera.com/blog/2014/12/new-in-cloudera-
labs-sparkonhbase/
45

3. Integration:
• Spark Cassandra Connector This library lets you
expose Cassandra tables as Spark RDDs, write Spark
RDDs to Cassandra tables, and execute arbitrary CQL
queries in your Spark applications. Supports also
integration of Spark Streaming with Cassandra
https://github.com/datastax/spark-cassandra-connector
• Spark + Cassandra using Deep: The integration
is not based on the Cassandra's Hadoop interface.
http://stratio.github.io/deep-spark/
• Getting Started with Apache Spark and Cassandra
http://planetcassandra.org/getting-started-with-apache-spark-and-cassandra/
• ‘Cassandra’ Tag at SparkBigData.com
http://sparkbigdata.com/component/tags/tag/20-cassandra
46

3. Integration:
• Benchmark of Spark & Cassandra Integration
using different approaches.
http://www.stratio.com/deep-vs-datastax/
• Calliope is a library providing an interface to consume
data from Cassandra to spark and store Resilient
Distributed Datasets (RDD) from Spark to Cassandra.
http://tuplejump.github.io/calliope/
• Cassandra storage backend with Spark is opening many new
avenues.
• Kindling: An Introduction to Spark with Cassandra
(Part 1) http://planetcassandra.org/blog/kindling-an-introduction-to-
spark-with-cassandra/
47

3. Integration:
• MongoDB is not directly served by Spark, although
it can be used from Spark via an official Mongo-
Hadoop connector.
• MongoDB-Spark Demo
https://github.com/crcsmnky/mongodb-spark-demo
• MongoDB and Hadoop: Driving Business Insights
http://www.slideshare.net/mongodb/mongodb-and-hadoop-driving-business-
insights
• Spark SQL also provides indirect support via its
support for reading and writing JSON text files.
https://github.com/mongodb/mongo-hadoop
48

3. Integration:
• There is also NSMC: Native Spark MongoDB Connector
for reading and writing MongoDB collections directly from
Apache Spark (still experimental)
• GitHub https://github.com/spirom/spark-mongodb-connector
• Using MongoDB with Hadoop & Spark
• https://www.mongodb.com/blog/post/using-mongodb-hadoop-spark-part-1-
introduction-setup PART 1
• http://www.mongodb.com/blog/post/using-mongodb-hadoop-spark-part-2-hive-
example Part 2
• http://www.mongodb.com/blog/post/using-mongodb-hadoop-spark-part-3-spark-
example-key-takeaways PART 3
• Interesting blog on Using Spark with MongoDB without
Hadoop
http://tugdualgrall.blogspot.fr/2014/11/big-data-is-hadoop-good-way-to-start.html
49

3. Integration:
• Neo4j is a highly scalable, robust (fully ACID), native graph
database.
• Getting Started with Apache Spark and Neo4j Using
Docker Compose. By Kenny Bastani, March 10, 2015
http://www.kennybastani.com/2015/03/spark-neo4j-tutorial-docker.html
• Categorical PageRank Using Neo4j and Apache Spark.
By Kenny Bastani, January 19, 2015
http://www.kennybastani.com/2015/01/categorical-pagerank-neo4j-spark.html
• Using Apache Spark and Neo4j for Big Data Graph
Analytics. By Kenny Bastani, November 3, 2014
http://www.kennybastani.com/2014/11/using-apache-spark-and-neo4j-for-big.html
50

3. Integration: YARN
• YARN: Yet Another Resource Negotiator, Implicit
reference to Mesos as the Resource Negotiator!
• Integration still improving.
https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%
20summary%20~%20yarn%20AND%20status%20%3D%20OPEN%20ORDER%20
BY%20priority%20DESC%0A
• Some issues are critical ones.
• Running Spark on YARN
http://spark.apache.org/docs/latest/running-on-yarn.html
• Get the most out of Spark on YARN
https://www.youtube.com/watch?v=Vkx-TiQ_KDU
51

3. Integration:
• Spark SQL provides built in support for Hive
tables:
• Import relational data from Hive tables
• Run SQL queries over imported data
• Easily write RDDs out to Hive tables
• Hive 0.13 is supported in Spark 1.2.0.
• Support of ORCFile (Optimized Row Columnar
file) format is targeted in Spark 1.3.0 Spark-2883
https://issues.apache.org/jira/browse/SPARK-2883
• Hive can be used both for analytical queries and
for fetching dataset machine learning algorithms
in MLlib.
52

3. Integration:
• Drill is intended to achieve the sub-second latency
needed for interactive data analysis and exploration.
http://drill.apache.org
• Drill and Spark Integration is work in progress in 2015 to
address new use cases:
• Use a Drill query (or view) as the input to Spark. Drill
extracts and pre-processes data from various data
sources and turns it into input to Spark.
• Use Drill to query Spark RDDs. Use BI tools to query
in-memory data in Spark. Embed Drill execution in a
Spark data pipeline.
Source: What's Coming in 2015 for
Drill?http://drill.apache.org/blog/2014/12/16/whats-coming-in-2015/
53

3. Integration:
• Apache Kafka is a high throughput distributed
messaging system. http://kafka.apache.org/
• Spark Streaming integrates natively with Kafka:
Spark Streaming + Kafka Integration Guide
http://spark.apache.org/docs/latest/streaming-kafka-integration.html
• Tutorial: Integrating Kafka and Spark Streaming:
Code Examples and State of the Game
http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-
example-tutorial/
• ‘Kafka’ Tag at SparkBigData.com
http://sparkbigdata.com/component/tags/tag/24-kafka
54

3. Integration:
• Apache Flume is a streaming event data
ingestion system that is designed for Big Data
ecosystem. http://flume.apache.org/
• Spark Streaming integrates natively with
Flume. There are two approaches to this:
• Approach 1: Flume-style Push-based Approach
• Approach 2 (Experimental): Pull-based
Approach using a Custom Sink.
• Spark Streaming + Flume Integration Guide
https://spark.apache.org/docs/latest/streaming-flume-integration.html
55

3. Integration:
• Spark SQL provides built in support for JSON that
is vastly simplifying the end-to-end-experience of
working with JSON data.
• Spark SQL can automatically infer the schema
of a JSON dataset and load it as a
SchemaRDD. No more DDL. Just point Spark
SQL to JSON files and query. Starting Spark 1.3,
SchemaRDD will be renamed to DataFrame.
• An introduction to JSON support in Spark SQL,
February 2, 2015 http://databricks.com/blog/2015/02/02/an-introduction-to-json-
support-in-spark-sql.html
56

3. Integration:
• Apache Parquet is a columnar storage format
available to any project in the Hadoop ecosystem,
regardless of the choice of data processing
framework, data model or programming language.
http://parquet.incubator.apache.org/
• Built in support in Spark SQL allows to:
• Import relational data from Parquet files
• Run SQL queries over imported data
• Easily write RDDs out to Parquet files
http://spark.apache.org/docs/latest/sql-programming-guide.html#parquet-files
• This is an illustrating example of integration of
Parquet and Spark SQL
http://www.infoobjects.com/spark-sql-parquet/
57

3. Integration:
• Spark SQL Avro Library for querying Avro data
with Spark SQL. This library requires Spark 1.2+.
https://github.com/databricks/spark-avro
• This is an example of using Avro and Parquet in Spark
SQL.
http://www.infoobjects.com/spark-with-avro/
• Avro/Spark Use case:
http://www.slideshare.net/DavidSmelker/bdbdug-data-types-jan-2015
• Problem
• Various inbound data sets
• Data Layout can change without notice
• New data sets can be added without notice
Result
• Leverage Spark to dynamically split the data
• Leverage Avro to store the data in a compact binary format
58

3. Integration: Kite SDK
• The Kite SDK provides high level abstractions to
work with datasets on Hadoop, hiding many of
the details of compression codecs, file formats,
partitioning strategies, etc.
http://kitesdk.org/docs/current/
• Spark support has been added to Kite 0.16
release, so Spark jobs can read and write to Kite
datasets.
• Kite Java Spark Demo
https://github.com/kite-sdk/kite-examples/tree/master/spark
59

3. Integration:
• Elasticsearch is a real-time distributed search and analytics
engine. http://www.elasticsearch.org
• Apache Spark Support in Elasticsearch was added in 2.1
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/spark.html
• Deep-Spark provides also an integration with Spark.
https://github.com/Stratio/deep-spark
• elasticsearch-hadoop provides native integration between
Elasticsearch and Apache Spark, in the form of RDD that can
read data from Elasticsearch. Also, any RDD can be saved to
Elasticsearch as long as its content can be translated into
documents. https://github.com/elastic/elasticsearch-hadoop
• Great use case by NTT Data integrating Apache
Spark Streaming and Elasticsearch.
http://www.intellilink.co.jp/article/column/bigdata-kk02.html
60

3. Integration:
• Apache Solr, added a Spark-based indexing tool for
fast and easy indexing, ingestion, and serving
searchable complex data. “CrunchIndexerTool on
Spark”
• Solr-on-Spark solution using Apache Solr, Spark,
Crunch, and Morphlines:
• Migrate ingestion of HDFS data into Solr from
MapReduce to Spark
• Update and delete existing documents in Solr at scale
• Ingesting HDFS data into Solr using Spark
http://www.slideshare.net/whoschek/ingesting-hdfs-
intosolrusingsparktrimmed
61

3. Integration:
• HUE is the open source Apache Hadoop Web UI
that lets users use Hadoop directly from their
browser and be productive. http://www.gethue.com
• A Hue application for Apache Spark called Spark
Igniter lets users execute and monitor Spark jobs
directly from their browser and be more
productive.
• Demo of Spark Igniter http://vimeo.com/83192197
• Big Data Web applications for Interactive Hadoop
https://speakerdeck.com/bigdataspain/big-data-web-applications-for-interactive-
hadoop-by-enrico-berti-at-big-data-spain-2014
62

1. Evolution
2. Transition
3. Integration
4. Complementarity
5. Key Takeaways
63

4. Complementarity
Components of Hadoop ecosystem and Spark ecosystem
can work together: each for what it is especially good at,
rather than choosing one of them.
64
Hadoop ecosystem Spark ecosystem

4. Complementarity: + +
• Tachyon is an in-memory distributed file system. By
storing the file-system contents in the main memory of all
cluster nodes, the system achieves higher throughput than
traditional disk-based storage systems like HDFS.
• The Future Architecture of a Data Lake: In-memory Data
Exchange Platform Using Tachyon and Apache Spark
(October 14, 2014)http://blog.pivotal.io/big-data-pivotal/news-2/the-future-
architecture-of-a-data-lake-in-memory-data-exchange-platform-using-tachyon-and-
apache-spark
• Spark and in-memory databases: Tachyon leading the
pack, January 2015
http://dynresmanagement.com/1/post/2015/01/spark-and-in-memory-databases-
tachyon-leading-the-pack.html
65

4. Complementarity: +
• Mesos and YARN can work together: each for what
it is especially good at, rather than choosing one of
the two for Spark deployment.
• Big data developers get the best of YARN’s power
for Hadoop-driven workloads, and Mesos’ ability
to run any other kind of workload, including non-
Hadoop applications like Web applications and other
long-running services.”
• Project Myriad is an open source framework for
running YARN on Mesos
• ‘Myriad’ Tag at SparkBigData.com
66

References:
• Apache Mesos vs. Apache Hadoop YARN
https://www.youtube.com/watch?v=YFC4-gtC19E
• Myriad: A Mesos framework for scaling a YARN
cluster https://github.com/mesos/myriad
• Myriad Project Marries YARN and Apache
Mesos Resource Management
http://ostatic.com/blog/myriad-project-marries-yarn-and-apache-mesos-
resource-management
• YARN vs. MESOS: Can’t We All Just Get
Along? http://strataconf.com/big-data-conference-ca-
2015/public/schedule/detail/40620
67

• Spark on Tez for efficient ETL:
https://github.com/hortonworks/spark-native-yarn
• Tez could takes care of the pure Hadoop optimization
strategies (building the DAG with knowledge of data
distribution, statistics or… HDFS caching).
• Spark execution layer could be leveraged without the
need of a nasty Spark/Hadoop coupling.
• Tez is good on fine-grained resource isolation with
YARN (resource chaining in clusters).
• Tez supports enterprise security.
68

• Data >> RAM: Processing huge data volumes,
much bigger than cluster RAM: Tez might be better,
since it is more “stream oriented” , has more mature
shuffling implementation, closer YARN integration.
• Data << RAM: Since Spark can cache in memory
parsed data, it can be much better when we process
data smaller than cluster’s memory.
• Improving Spark for Data Pipelines with Native
YARN Integration http://hortonworks.com/blog/improving-spark-data-
pipelines-native-yarn-integration/
• Get the most out of Spark on YARN
https://www.youtube.com/watch?v=Vkx-TiQ_KDU
69

4. Complementarity
• Emergence of the ‘Smart Execution Engine’ Layer:
Smart Execution Engine dynamically selects the optimal
compute framework at each step in the big data
analytics process based on the type of platform, the
attributes of the data and the condition of the cluster.
• Matt Schumpert on Datameer Smart Execution Engine
http://www.infoq.com/articles/datameer-smart-execution-engine Interview on
November 13, 2014 with Matt Schumpert, Director of Product
Management at Datameer.
• The Challenge to Choosing the “Right” Execution
Engine. By Peter Voss | September 30, 2014
http://www.datameer.com/blog/announcements/the-challenge-to-choosing-the-
right-execution-engine.html
70

4. Complementarity
• Operating in a Multi-execution Engine Hadoop Environment by
Erik Halseth of Datameer on January 27th, 2015 at the Los Angeles
Big Data Users Group.
• http://files.meetup.com/12753252/LA Big Data Users Group Presentation Jan-27-
2015.pdf
• New Syncsort Big Data Software Removes Major Barriers to
Mainstream Apache Hadoop Adoption, February 12, 2015
http://www.itbusinessnet.com/article/New-Syncsort-Big-Data-Software-
Removes-Major-Barriers-to-Mainstream-Apache-Hadoop-Adoption-3749366
• Syncsort Automates Data Migrations Across Multiple Platforms,
February 23, 2015
http://www.itbusinessedge.com/blogs/it-unmasked/syncsort-automates-data-
migrations-across-multiple-platforms.html
• Framework for the Future of Hadoop, March 9, 2015
http://blog.syncsort.com/2015/03/framework-future-hadoop/
71

5. Key Takeaways
1. Evolution: of compute models is still ongoing.
Watch out Apache Flink project for true low-
latency and iterative use cases!
2. Transition: Tools from the Hadoop ecosystem
are still being ported to Spark. Keep watching
general availability and balance risk and
opportunity.
3. Integration: Healthy dose of Hadoop ecosystem
integration with Spark. More integration is on
the way.
4. Complementarity: Components and tools from
Hadoop ecosystem and Spark ecosystem can
work together: each for what it is especially good
at. One size doesn’t fit all!
72

1. File System
2. Deployment
3. Distributions
4. Alternatives
5. Key Takeaways
73

1. File System
Spark does not require HDFS: Hadoop Distributed File System! Your
‘Big Data’ use case might be implemented without HDFS! For example:
1. Use Spark to process data stored in Cassandra File System (DataStax
CassandraFS) or MongoDB File System (GridFS)
2. Use Spark to read and write data directly to a messaging system like
Kafka if your use case doesn’t need data persistence. Example:
http://techblog.netflix.com/2015/03/can-spark-streaming-survive-chaos-monkey.html
3. Use an In-Memory distributed File System such as Spark’s cousin:
Tachyon http://sparkbigdata.com/component/tags/tag/13
4. Use a Non-HDFS file system’ already supported by Spark:
• Amazon S3
• http://databricks.gitbooks.io/databricks-spark-reference-
applications/content/logs_analyzer/chapter2/s3.html
• MapR-FS
• https://www.mapr.com/blog/comparing-mapr-fs-and-hdfs-nfs-and-snapshots
5. OpenStack Swift (Object Store)
• https://spark.apache.org/docs/latest/storage-openstack-swift.html
• https://www.openstack.org/summit/openstack-paris-summit-2014/session-
videos/presentation/the-perfect-match-apache-spark-meets-swift
74

1. File System
When coupled with its analytics capabilities, file-
system agnostic Spark can only re-ignite this
discussion of HDFS alternatives. Because Hadoop isn’t
perfect: 8 ways to replace HDFS. July 11, 2012
https://gigaom.com/2012/07/11/because-hadoop-isnt-perfect-8-ways-to-
replace-hdfs/
A few HDFS alternatives to choose from, include:
• Apache Spark on Mesos running on CoreOS and using EMC ECS
HDFS storage. March 9, 2015
http://www.recorditblog.com/post/apache-spark-on-mesos-running-
on-coreos-and-using-emc-ecs-hdfs-storage/
• Lustre File System - Intel Enterprise Edition for Lustre (IEEL)
(Upcoming support)
http://insidebigdata.com/2014/10/02/replacing-hdfs-lustre-maximum-
performance/
• Quantcast QFS https://www.quantcast.com/engineering/qfs
• …
75

1. File System
2. Deployment
3. Distributions
4. Alternatives
5. Key Takeaways
76

2. Deployment
While Spark is most often discussed as a replacement for MapReduce
in Hadoop clusters to be deployed on YARN, Spark is actually
agnostic to the underlying infrastructure for clustering, so
alternative deployments are possible:
1. Local: http://sparkbigdata.com/tutorials/51-deployment/121-local
2. Standalone: http://sparkbigdata.com/tutorials/51-deployment/123-standalone
3. Apache Mesos: http://sparkbigdata.com/tutorials/51-deployment/122-mesos
4. Amazon EC2: http://sparkbigdata.com/tutorials/51-deployment/124-amazon-ec2
5. Amazon EMR: http://sparkbigdata.com/tutorials/51-deployment/127-amazon-emr
6. Rackspace: http://sparkbigdata.com/tutorials/51-deployment/138-on-rackspace
7. Google Cloud Platform:http://sparkbigdata.com/tutorials/51-deployment/139-
google-cloud
8. HPC Clusters:
• Setting up Spark on top of Sun/Oracle Grid Engine (PSI) -
http://sparkbigdata.com/tutorials/51-deployment/126-sun-oracle-grid-engine-sge
• Setting up Spark on the Brutus and Euler Clusters (ETH) -
http://sparkbigdata.com/tutorials/51-deployment/128-hpc-cluster
77

1. File System
2. Deployment
3. Distributions
4. Alternatives
6. Key Takeaways
78

3. Distributions
• Using Spark on a Non-Hadoop distribution:
79

Cloud
• Databricks Cloud is not dependent on
Hadoop. It gets its data from Amazon’s S3
(most commonly), Redshift, Elastic MapReduce.
https://databricks.com/product/databricks-cloud
• Databricks Cloud: From raw data, to insights and
data products in an instant! March 4, 2015
https://databricks.com/blog/2015/03/04/databricks-cloud-from-raw-data-to-
insights-and-data-products-in-an-instant.html
• Databricks Cloud Announcement and Demo at
Spark Summit 2014, July 2, 2014
https://www.youtube.com/watch?v=dJQ5lV5Tldw
80

DSE:
• DSE: DataStax Enterprise built on Apache Cassandra
presents itself as a Non-Hadoop Big Data Platform.
Data can be stored in Cassandra File System.
http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enter
prise/spark/sparkTOC.html
• Escape from Hadoop: Ultra Fast Data Analysis with
Spark & Cassandra, Piotr Kolaczkowski, September 26, 2014
http://www.slideshare.net/PiotrKolaczkowski/fast-data-analysis-with-spark-4
• Escape from Hadoop: with Apache Spark and
Cassandra with the Spark Cassandra Connector
Helena Edelson, published on November 24, 2014
http://www.slideshare.net/helenaedelson/escape-from-hadoop-with-apache-
spark-and-cassandra-41950082
81

• Stratio is a Big Data platform based on Spark. It
is 100% open source and enterprise ready
http://www.stratio.com
• Streaming-CEP-Engine: Streaming CEP engine
is a Complex Event Processing platform built
on Spark Streaming. It is the result of combining
the power of Spark Streaming as a continuous
computing framework and Siddhi CEP engine as
complex event processing engine.
http://stratio.github.io/streaming-cep-engine/
• ‘Stratio’ Tag at SparkBigData.com
82

83
• xPatterns (http://atigeo.com/technology/) is a complete big
data analytics platform available with a novel
architecture that integrates components across
three logical layers: Infrastructure, Analytics,
and Applications.
• xPatterns is cloud-based, exceedingly scalable,
and readily interfaces with existing IT systems.
• ‘xPatterns’ Tag at
SparkBigData.comhttp://sparkbigdata.com/component/tags/tag/
39

84
• The BlueData (http://www.bluedata.com/) EPIC software
platform solves the infrastructure challenges and
limitations that can slow down and stall Big Data
deployments.
• With EPIC software, you can spin up Hadoop
clusters – with the data and analytical tools that
your data scientists need – in minutes rather than
months. https://www.youtube.com/watch?v=SE1OP4ImrxU
• ‘BlueData’ Tag at SparkBigData.com

85
• Guavus (http://www.guavus.com) embeds Apache Spark into
its Operational Intelligence Platform Deployed at the
World’s Largest Telcos. September 25, 2014 by Eric Carr
http://databricks.com/blog/2014/09/25/guavus-embeds-apache-spark-into-its-
operational-intelligence-platform-deployed-at-the-worlds-largest-telcos.html
• Guavus operational intelligence platform analyzes
streaming data and data at rest.
• The Guavus Reflex 2.0 platform is commercially
compatible with open source Apache Spark.
http://insidebigdata.com/2014/09/26/guavus-databricks-announce-reflex-
platform-now-certified-spark-distribution/
• ‘Guavus’ Tag at SparkBigData.com

1. File System
2. Deployment
3. Distributions
4. Alternatives
5. Key Takeaways
86

4. Alternatives
Hadoop Ecosystem Spark Ecosystem
Component
HDFS Tachyon
YARN Mesos
Tools
Pig Spark native API
Hive Spark SQL
Mahout MLlib
Storm Spark Streaming
Giraph GraphX
HUE Spark Notebook/ISpark
87


• Tachyon is a memory-centric distributed file
system enabling reliable file sharing at memory-
speed across cluster frameworks, such as Spark
and MapReduce. https://http://tachyon-project.org
• Tachyon is Hadoop compatible. Existing Spark
and MapReduce programs can run on top of it
without any code change.
• Tachyon is the storage layer of the Berkeley
Data Analytics Stack (BDAS)
https://amplab.cs.berkeley.edu/software/
88


• Mesos (http://mesos.apache.org/) enables fine
grained sharing which allows a Spark job to dynamically
take advantage of the idle resources in the cluster during
its execution. This leads to considerable performance
improvements, especially for long running Spark jobs.
• Mesos as Data Center “OS”:
• Share datacenter between multiple cluster computing
apps; Provide new abstractions and services
• Mesosphere DCOS: Datacenter services, including
Apache Spark, Apache Cassandra, Apache YARN,
Apache HDFS…
• ‘Mesos’ Tag at SparkBigData.com
http://sparkbigdata.com/component/tags/tag/16-mesos
89

YARN vs. Mesos
Criteria
Resource
sharing
Yes Yes
Written in Java C++
Scheduling Memory only CPU and Memory
Running tasks Unix processes Linux Container groups
Requests Specific requests
and locality
preference
More generic but more
coding for writing
frameworks
Maturity Less mature Relatively more mature
90

 Spark Native API
• Spark Native API in Scala, Java and Python.
• Interactive shell in Scala and Python.
• Spark supports Java 8 for a much more concise
Lambda expressions to get code nearly as
simple as the Scala API.
• ETL with Spark - First Spark London Meetup,
May 28, 2014
http://www.slideshare.net/rafalkwasny/etl-with-spark-first-spark-london-
meetup
• ‘Spark Core’ Tag at
SparkBigData.comhttp://sparkbigdata.com/component/tags/tag/
11-core-spark
91

 Spark SQL
• Spark SQL is a new SQL engine designed from
ground-up for Spark: https://spark.apache.org/sql/
• Spark SQL provides SQL performance and maintains
compatibility with Hive. It supports all existing Hive data
formats, user-defined functions (UDF), and the Hive
metastore.
• Spark SQL also allows manipulating (semi-) structured
data as well as ingesting data from sources that
provide schema, such as JSON, Parquet, Hive, or
EDWs. It unifies SQL and sophisticated analysis,
allowing users to mix and match SQL and more
imperative programming APIs for advanced analytics.
92

 Spark MLlib
93
‘Spark MLlib ’ Tag at
SparkBigData.comhttp://sparkbigdata.com/component/tags/tag/5-mllib

 Spark Streaming
94
‘Spark Streaming ’ Tag at http://sparkbigdata.com/component/tags/tag/3-
spark-streaming

Storm vs. Spark Streaming
Criteria
Processing Model Record at a time Mini batches
Latency Sub second Few seconds
Fault tolerance–
every record
processed
At least one ( may
be duplicates)
Exactly one
Batch Framework
integration
Not available Core Spark API
Supported
languages
Any programming
language
Scala, Java,
Python
95

 GraphX
96
‘GraphX’ Tag at
SparkBigData.comhttp://sparkbigdata.com/component

 Notebook
97
• Zeppelin http://zeppelin-project.org, is a web-based
notebook that enables interactive data analytics.
Has built-in Apache Spark support.
• Spark Notebook is an interactive web-based
editor that can combine Scala code, SQL
queries, Markup or even JavaScript in a
collaborative manner. https://github.com/andypetrella/spark-
notebook
• ISpark is an Apache Spark-shell backend for
IPython https://github.com/tribbloid/ISpark

IV. Spark on Non-Hadoop
1. File System
2. Deployment
3. Distributions
4. Alternatives
5. Key Takeaways
98

6. Key Takeaways
1. File System: Spark is File System Agnostic.
Bring Your Own Storage!
2. Deployment: Spark is Cluster Infrastructure
Agnostic. Choose your deployment.
3. Distributions: You are no longer tied to Hadoop
for Big Data processing. Spark distributions as
service in the cloud or imbedded in Non-Hadoop
distributions are emerging!
4. Alternatives: Do your due diligence based on
your own use case and research pros and cons
before picking a specific tool or switching from one
tool to another.
99

IV. More Q&A
100
http://www.SparkBigData.com
sbaltagi@gmail.com
https://www.linkedin.com/in/slimbal
tagi
@SlimBaltagi
http://www.slideshare.net/sbaltagi

Spark or Hadoop: Is it an either-or proposition

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to Spark or Hadoop: Is it an either-or proposition

Similar to Spark or Hadoop: Is it an either-or proposition (20)

More from Slim Baltagi

More from Slim Baltagi (13)

Recently uploaded

Recently uploaded (20)

Spark or Hadoop: Is it an either-or proposition