1. Airflow

2. Open source platform to programmatically author
and monitor data pipelines.
Airflow

3. Street Cred

4. * Companies grow to have a complex network of
processes that have intricate dependencies
* Analytics & batch processing are mission
critical
* Tons of time is spent writing jobs, monitoring
and troubleshooting issues
The Problem Statement

5. * Data lineage is opaque
* Learning curve gets steeper as the ecosystem
grows
* Code / logic is duplicated (not OO)
* Open ended framework = heterogenous methods
* Ownership and accountability falls behind
* Maintenance time grows exponentially
* Operational metadata (job duration, landing times,
logs, …) is scattered if existent
* Loose guidelines leads to bad practices
* Signal to noise ratio on alerts gets out of control
Common Symptoms

6. * Airbnb was outgrowing its workflow
methodology
* High hopes:
* Dynamic pipeline generation
* Programmatic environment
* A platform people love working with
* We took it as an opportunity to build and share
Genesis

7. Airflow@Airbnb
Data Warehousing
Experimentation
Growth Analytics
Email Targeting
Sessionization
Search Ranking
Infrastructure Maintenance
Engagement Analytics
Anomaly Detection

8. Scale
* 6 * c3.8xlarge nodes * 128 processing slots
* 5-6k tasks per day
* 30 pipeline authors @Airbnb
* 8 Airflow contributors
* 5 companies using Airflow in production
* No bottleneck in sight

9. * Python!
* A focus on authoring
* Programmatic: the DAG definition is code
* Modular & easily extensible
* Rich CLI
* Rich web UI
Design Choices

11. UI Demo!

21. * Python backend
* Distributed execution is extensible, but we use
Celery on Redis for now
* UI: Flask / SqlAlchemy / d3.js / Highcharts
* Templating: Jinja!
* Metadata database: SqlAlchemy (MySQL)
* Airflow/requirements.txt
Technologies

22. * BashOperator
* PythonOperator
* HiveOperator
* MySqlOperator
* CascadingOperator
* SparkOperator
* DummyOperator
* EmailOperator
* ExternalTaskSensor
* HdfsSensor
* Hive2SambaOperator
* HivePartitionSensor
Operators are Task Factories
* MySqlToHiveTransfer
* PostgresOperator
* PrestoCheckOperator
* PrestoIntervalCheckOperator
* PrestoValueCheckOperator
* HiveToMySqlTransfer
* S3KeySensor
* S3ToHiveTransfer
* SqlSensor
* TimeSensor
* MORE! …

23. Local Repo
Master Repo
Metadata
Database
…
Workers
MySQL
Hive
HDFS
Web Servers
Cascading
Scheduler
Spark
Presto
…
Architecture
Web ServersWeb ServersWeb ServersWeb Servers

25. Conclusion

26. In the works / next steps
* Build a community!
* class YarnExecutor(BaseExecutor):
* Sharing our services / frameworks
* Hive stats collection
* Anomaly detection
* Experimentation framework

28. Qs?