An Apache Spark introduction I presented at Budapest Data Forum 2015.

1. Apache Spark
Mate Gulyas

2. WHY WE DO IT?
54% average
viewability**
36% non-human
visitor *
What percentage of
digital ads reach
people?
Clickbots,
botnets
Invisible,
hidden ads
Transparency in
the market
*http://technorati.com/iab-keynote-36-percent-ad-traffic-from-bots-and-threatening-industry/
**http://www.statista.com/statistics/255061/viewability-rates-for-rich-media-ads-worldwide-by-industry/

3. JavaScript SegmentationBehaviour analysis
WHAT WE DO?

4. Distributed data processing
WHAT DO WE NEED?
Averge size client
30 GB / day
900 GB / month
20 average size clients
600 GB / day
18 TB / month

5. Recurring data
transformations
WHAT DO WE NEED?

6. Interactive / Batch /
Streaming / SQL /
Graph processing
IT’S SO COOL

7. In-memory
WHY SPARK?

8. Productive API
WHY SPARK?

9. Multiple language
WHY SPARK?

10. Active community
WHY SPARK?

11. friendly
WHY SPARK?
developer
analyst
CFO

12. friendly
WHY SPARK?
developer
analyst
CFO

13. friendly
WHY SPARK?
developer
analyst
CFO

14. Resilient Distributed
Dataset (RDD)
ONE THING TO REMEMBER

15. RDD

16. IT RUN’S ON
Mesos
YARN
Standalone
AWS EC2

17. IT GET’S DATA FROM?
Amazon S3, HDFS,
Cassandra, Hive, Hbase,
Tachyon, Local Filesystem,
ODBC databases, etc...

18. Batch processing
THE OLD WAY

19. Interactive analytics
THE NEW WAY

20. SPARK WITH IPYTHON

21. Spark SQL
THE NOT THAT OLD WAY

22. {"name": "Mate Gulyas", "twitter": "gulyasm"}
{"name": "John Doe", "email": "jdoe@freemail.com"}
{"name": "Jane Doe", "email": "janedoe@citromail.com"}
val input = hiveCtx.jsonFile(“example.json”)
input.registerAsTable(“users”)
hiveCtx.sql(“SELECT name, twitter FROM people;”)
SQL WITH JSON

23. Spark Streaming
THE LOW LATENCY WAY

24. DStream

25. MLlib
THE SKYNET WAY

26. GraphX
I LOVE GRAPHS

27. Third party modules
THE OTHERS WAY

28. On-premises
AWS
Databricks Cloud
BUT… WHERE TO GO?

29. TAKEAWAY I
Spark can provide one
platform to cover most of
the use-cases in data
analytics

30. TAKEAWAY II
Productive, fast data
processing framework that
helps you minimize to time
business impact.

31. MATE GULYAS
gulyasm@enbrite.ly
@gulyasm
@enbritely
THANK YOU!