Are you a Java developer interested in big data processing and never had the chance to work with Apache Spark ? My presentation aims to help you get familiar with Spark concepts and start developing your own distributed processing application.

2. Spark –
The beginnings
Daniel Leon
Optymyze
[7th of November 2015]

3. Content
1) Hadoop Dilemma
2) Processing engines war
3) Spark ecosystem
4) Resilient Distributed Datasets
5) Spark application workflow
6) Conclusion

4. Big data technologies

5. Hadoop – Where does it end ?

6. Hadoop Architecture

7. Hadoop evolution

8. Map Reduce workflow

9. Hadoop ecosystem

10. Beyond Map Reduce
•Complex iterative algorithms
•Interactive queries
•Real time processing

11. Different processing model
•More operation available
•Flexible way of composing operations
•Pluggable data sources
•Streaming capabilities built-in
•Pluggable algorithm

12. Searching for another
processing engine

13. Processing engine comparison

14. Processing engine comparison

15. Spark ecosystem

16. Spark ecosystem

17. 100TB Daytona Sort Competition
2014

18. Resilient Distributed Dataset -
RDD
•Stored in memory and storage
•Immutable
•Enables parallel operations on collections of
elements
•Contains lineage information

19. Resilient Distributed Dataset -
RDD

20. Constructing RDD's
•Parallelize existing collections
lRDD=sc.parallelize([“a”, “b”, “c”])
•From files in HDFS, S3, Hive
llinesRDD=sc.textFile(“README”)
•Transforming an existing RDD

21. Operations on RDD's
•Transformations – lazy
lfilter
lmap
lgroupBy
•Actions
lcount
lcollect

22. Spark terminology
•Job – the work required to compute an RDD
•Stage – a wave of work within a job,
corresponding to one or more pipelined
RDD's
•Task – a unit of work within a stage,
correspoding to one RDD partition
•Shuffle – the transfer the data between stages

23. Spark architecture

24. Conclusion
•Spark is :
•Complete and standalone solution for
distributed processing
•Fluent API
•Pluggable with other big data frameworks
•One of the most actively contributed Apache
project

25. Documentation
https://hadoopecosystemtable.github.io
https://databricks.com/spark/developer-resources
https://databricks.com/resources/slides
https://databricks.com/spark/training

26. Spark – The beginnings
Daniel Leon
Optymyze
[7th of November 2015]