Learn how to use PySpark for processing massive amounts of data. Combined with the GitHub repo - https://github.com/rdempsey/pyspark-for-data-processing - this presentation will help you gain familiarity with processing data using Python and Spark. If you're thinking about machine learning and not sure if it can help improve your business, but want to find out, set up a free 20-minute consultation with us: https://calendly.com/robertwdempsey/free-consultation

1. Using PySpark to Process
Boat Loads of Data
Robert Dempsey, CEO
Atlantic Dominion Solutions

3. We’ve mastered three jobs so you can
focus on one - growing your business.

4. The Three Jobs
At Atlantic Dominion Solutions we perform three functions for
our customers:
Consulting: we assess and advise in the areas of technology,
team and process to determine how machine learning can have
the biggest impact on your business.
Implementation: after a strategy session to determine the work
you need we get to work using our proven methodology and
begin delivering smarter applications.
Training: continuous improvement requires continuous learning.
We provide both on-premises and online training.

5. Co-authoring the book Building
Machine Learning Pipelines.
Written for software developers and
data scientists, Building Machine
Learning Pipelines teaches the skills
required to create and use the
infrastructure needed to run
modern intelligent systems.
machinelearningpipelines.com
Writing the Book

6. Robert Dempsey, CEO
Software Engineer
Books and online courses
Lotus Guides, District Data Labs
Atlantic Dominion Solutions, LLC
Professional
Author
Instructor
Owner

7. What You Can Expect
Today

8. MTAC Framework™
Mindset
Toolbox
Application
Communication

9. 1. When acquiring knowledge start by going wide instead of
deep.
2. Always focus on what's important to people rather than
just the technology.
3. Be able to clearly communicate what you know with
others.
Core Principles

10. MTAC Framework™ Applied
Mindset: use-case centric example
Toolbox: Python, PySpark, Docker
Application: Code & Analysis
Communication: Q&A

11. Mindset

12. Keep It Simple
Image: Jesse van Dijk : http://jessevandijkart.com/the-labyrinth-of-tsan-kamal/

13. Solve the Problem
Image: Paulo : https://paullus23.deviantart.com/art/Bliss-soccer-field-326563199

14. Explain It, Simply

15. Break Through

16. Use Case

17. Got Clean Air?

18. Got Clean Air?
• Clean air is important.
• Toxic pollutants are known or suspected of causing cancer,
reproductive effects, birth defects, and adverse
environmental effects.

19. Questions to Answer
1. Which state has the highest level of pollutants?
2. Which county has the highest level of pollutants?
3. What are the top 5 pollutants by unit of measure?
4. What are the trends of pollutants by state over time?

20. Toolbox

21. Python

23. Spark

24. The Core of Spark
• Computational engine that schedules, distributes and
monitors computational tasks running on a cluster

25. Higher Level Tools
• Spark SQL: SQL and structured data
• MLlib: machine learning
• GraphX: graph processing
• Spark Streaming: process streaming data

26. Storage
• Local file system
• Amazon S3
• Cassandra
• Hive
• HBase
• File formats
• Text files
• Sequence files
• Avro
• Parquet
• Hadoop Input Format

27. Hadoop?
• Not necessary, but…
• If you have multiple nodes you need a resource manager
like YARN or Mesos
• You'll need access to distributed storage like HDFS,
Amazon S3 or Cassandra

28. PySpark

29. What Is PySpark?
• An API that exposes the Spark programming model to
Python
• Build on top of Spark's Java API
• Data is processed with Python and cached/shuffled in the
JVM
• Driver programs

30. Driver Programs
• Launch parallel operations on a cluster
• Contain application functions
• Define distributed datasets
• Access Spark through a SparkContext
• Uses Py4J to launch a JVM and create a
JavaSparkContext

31. When to Use It
• When you need to…
• Process boat loads of data (TB)
• Perform operations that require all the data to be in
memory (machine learning)
• Efficiently process streaming data
• Create an overly complicated use case to present at a
meetup

32. Docker

33. Docker
• Software container platform
• Containers are application only (no OS)
• Deployed anywhere with same CPU architecture (x86-64,
ARM)
• Available for *nix, Mac, Windows

34. Container Architecture

35. Application

36. PySpark in
Data Architectures

37. Architecture #1
Agent
File
System
Apache
Spark
File
System
Agent ES
1 2 3
Data Flow

38. Architecture #2
Data Flow
Agent
1 2 3
Agent
Agent
Athena
S3
S3
Apache
Spark

39. Architecture #3
Data Flow
Agent
1 2 3
Agent
Agent
ES
S3
HDFS
Apache
Kafka
Apache
Spark
HBase

40. What We’ll Build (Simple)
Agent
File
System
Apache
Spark
File
System
1 2 3
Data Flow

41. Python
• Analysis
• Visualization
• Code in our Spark jobs

42. Spark
• By using PySpark

43. PySpark
• Process all the data!
• Perform aggregations

44. Docker
• Run Spark in a Docker container.
• So you don’t have to install anything.

45. Code Time!

46. README
• https://github.com/rdempsey/pyspark-for-data-processing
• Create a virtual environment (Anaconda)
• Install dependencies
• Run docker-compose to create the Spark containers
• Run a script (or all of them!) per the README

47. Dive In
• Data explorer notebook
• Q1 - Most polluted state
• Q2 - Most polluted county
• Q3 - Top pollutants by unit of measure
• Q4 - Pollutants over time

48. Communication

49. Q&A

50. Early Bird Specials!

51. Intro to Data Science for
Software Engineers
Goes live October 23, 2017
Normally: $97
Pre-Launch: $47
http://lotusguides.com

52. Where to Find Me
Website
Lotus Guides
LinkedIn
Twitter
Github
robertwdempsey.com
lotusguides.com
robertwdempsey
rdempsey
rdempsey

53. Thank You!