Presentation on Spark SQL + Streaming for Philly JUG

1. Spark
Brian O’Neill (@boneill42)
Monetate

2. Agenda
● History / Context
○Hadoop
○Lambda
●Spark Basics
○RDDs, Dataframe, SQL, Streaming
● Play along / Demo

3. We work at Monetate...
Client
(e.g. Retailer)
Decision
Engine
Data
Analytics
Engine
consumer marketer
Dashboard
Warehouse
Meta
Observations

4. We call it a...
Personalization Platform
Not so hard until...
m’s → B’s
100ms’s → 10ms’s
days → minutes
(sessions / month)
(response times)
(analytics lag)

5. HISTORY

6. history - hadoop

7. map / reduce
tuple = (key, value)
map(x) -> tuple[]
reduce(key, value[]) -> tuple[]

8. word count
The Code
def map(doc)
doc.each do |word|
emit(word, 1)
end
end
def reduce(key, values[])
sum = values.inject {|sum,x| sum + x }
emit(key, sum)
end
The Run
doc1 = “boy meets girl”
doc2 = ”girl likes boy”)
map (doc1) -> (boy, 1), (meets, 1), (girl, 1)
map (doc2) -> (girl, 1), (likes, 1), (boy, 1)
reduce (boy, [1, 1]) -> (boy, 2)
reduce (girl, [1, 1]) -> (girl, 2)
reduce (likes [1]) -> (likes, 1)
reduce (meets, [1]) -> (meets, 1)

9. Jobs on top of jobs...

10. Real-time? Different hammer.

11. Let’s invent some terminology...

12. Traditional lambda...

13. Can we collapse the lambda?

14. Spark- FTW!

15. Lambda on Spark (e.g.)
S3
Kafka
MySQL
RDD
RDD
Dataframe
Druid

16. SPARK BASICS

17. Concept : RDDs
“Spark revolves around the concept of a resilient distributed
dataset (RDD), which is a fault-tolerant collection of
elements that can be operated on in parallel. There are two
ways to create RDDs: parallelizing an existing collection in
your driver program, or referencing a dataset in an external
storage system, such as a shared filesystem, HDFS,
HBase, or any data source offering a Hadoop InputFormat.”
http://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds

18. Concept : Transformations &
Operations
Transformation:
RDD(s) → RDD
e.g. map, filter, groupBy, etc.
Action:
RDD → value
e.g. reduce, count, etc.

19. Code: RDDs
JavaPairRDD<Integer, Product> productsRDD = javaFunctions(sc)
.cassandraTable("java_api", "products", productReader)
.keyBy(new Function<Product, Integer>() {
@Override
public Integer call(Product product) throws Exception {
return product.getId();
}
});

20. DAGs
Lazily evaluated!

21. Concept : DataFrames
DataFrames = RDD + Schema
“A DataFrame is a distributed collection of data organized
into named columns. It is conceptually equivalent to a table
in a relational database or a data frame in R/Python, but
with richer optimizations under the hood. DataFrames can
be constructed from a wide array of sources such as:
structured data files, tables in Hive, external databases, or
existing RDDs.”
http://spark.apache.org/docs/latest/sql-programming-guide.html#dataframes

22. Concept : Spark SQL
SELECT
min(event_time) AS start_time,
max(event_time) AS end_time,
account_id
FROM events GROUP BY account_id

23. Code: SQL + Dataframes
StructType schema = configuration.getSchemaForProduct();
DataFrame dataFrame = sqlContext.createDataFrame(productsRDD, schema);
sqlContext.registerDataFrameAsTable(dataFrame, “products”);

24. And remember Uncle Ben…
“With great power, comes great responsibility.”

25. Concept : Streaming
.forEachRDD

26. Code: Streaming
JavaStreamingContext streamingContext = new JavaStreamingContext(getSparkConf(),
SessionizerState.getConfig().getSparkStreamingBatchDuration());
JavaReceiverInputDStream<byte[]> kinesisStream = KinesisUtils.createStream(...);
kinesisStream.foreachRDD(new VoidFunction<JavaRDD<byte[]>>() {
@Override
public void call(JavaRDD<byte[]> rdd) throws Exception {
JavaRDD<String> lines = rdd.map(new Function<byte[], String>( {
public String call(byte[] bytes) throws IOException {
return new String(bytes, Charset.forName("UTF-8"));
}
});
processRdd(lines);
}
});

27. DEPLOYMENT

28. Basic Architecture
http://spark.apache.org/docs/latest/cluster-overview.html

29. Kinesis / Streaming Architecture

30. Amazon’s EMR

31. Play along demo.

32. Get stuff...
Get Spark...
http://spark.apache.org/downloads.html
Get Cassandra…
http://cassandra.apache.org/download/
Get Code…
https://github.com/boneill42/spark-on-cassandra-quickstart

33. Configure stuff...
$spark/conf> cp spark-env.sh.template spark-env.sh
$spark/conf> echo "SPARK_MASTER_IP=127.0.0.1" >> spark-env.sh

34. Start stuff...
# Start Master
$spark> sbin/start-master.sh
$spark> tail -f logs/*
# Start Worker
$spark> bin/spark-class org.apache.spark.deploy.worker.Worker
spark://127.0.0.1:7077

35. Build and launch stuff...
# Build
$code> mvn clean install
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
# Launch
$code> spark-submit --class com.github.boneill42.JavaDemo
--master spark://127.0.0.1:7077
target/spark-on-cassandra-0.0.1-SNAPSHOT-jar-with-dependencies.jar
spark://127.0.0.1:7077 127.0.0.1

36. A message from our
sponsor

37. Advertisements...
https://github.com/monetate/koupler
https://github.com/monetate/ectou-metadata
https://github.com/monetate/ectou-export