Like many software projects, data pipelines built by Business Intelligence teams often start out as quickly built monoliths, but over time they can be made simpler and more maintainable by splitting them into a series of loosely coupled jobs, stealing ideas from service oriented architecture. In this presentation we'll look at the challenges we've faced scaling Data Analytics at Hootsuite, then move into a live coding session, where we'll stitch together a data pipeline as a series of Scala apps, deployed to AWS Lambda, connected using Airbnb's open source Airflow tool.

1. Let’s Build a Service Oriented
Data Pipeline!
June 2016
Software Developer | Hootsuite
Yasha Podeswa

2. Before: Oceanographer
Me!

3. Now: Software Developer at Hootsuite
Me!

4. Introduce a problem that requires a new data pipeline
Design it in a service oriented style
Build it on stage!
This Talk

5. Passive Aggressive Inc. just
cancelled their subscription!
Desperate Dan in trouble!
The Problem

6. Want to Build a Tool Like This

7. Want to Build a Tool Like This

8. Want to Build a Tool Like This

10. What We’re Starting With

11. What We’re Starting With
Things Users
Did

12. What We’re Starting With
Things
Organizations
Did

13. What We’re Starting With
Crap

14. High Level Plan
JSON files
Calculate stats
about
organizations
DB

15. High Level Plan
JSON files
Calculate stats
about
organizations
DB
Extract
Transform
Load

16. High Level Plan
JSON files
Calculate stats
about
organizations
DB
Extract
Transform
Load

17. JSON files
Calculate stats
about
organizations
DB
Clean and organize
data
Calculate stats per
organization

18. JSON files
Calculate stats
about
organizations
DB
Clean and organize
data
Calculate stats per
organization
Useful for lots of
things!

19. JSON files
Calculate stats
about
organizations
DB
Clean and organize
data
Calculate stats per
organization
Shouldn’t run until
dependent job done

20. Need a “Service” Communication and
Orchestration Layer!

22. Let’s build it!

23. First App Event Cleaning and Loading
Read logs from S3, clean and sort into different types of
events, load into data warehouse
Vanilla Scala app
AWS Lambda

24. Second App Organization Stat Calculation
Read cleaned/sorted events from data warehouse, calculate
stats about organization, load stats to data warehouse
Vanilla Scala app
AWS Lambda

25. Third App Airflow
Hook up the Lambda apps in a dependency graph
● Scheduling
● Retries
● Monitoring

26. Steal my code!
https://github.com/yashap/etl-load-events
https://github.com/yashap/etl-organization-stats
https://github.com/yashap/airflow

27. Questions?