We will cover some resources for getting started with Airflow. Airflow is a python based scheduling tool with the ability to connect to a number of different data management tools. Accompanying GitHub: https://github.com/anomnaco/WorkshopScratchpad Accompanying Blog: https://blog.anant.us/data-engineers-lunch-13-introduction-to-airflow/ Accompanying YouTube: https://youtu.be/9SYhd2yfE_4 Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday: https://www.meetup.com/Data-Wranglers-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ #airflow #python

1. Version 1.0
Introduction to Airflow
An Anant Corporation Story.

2. Airflow Overview
● A tool for scheduling and automating workflows and tasks
● Good for automating repeated processes
○ Common ETL tasks
○ Machine learning model training
● Write workflows in Python
○ Tools interactable via Python should work as well
○ Define dependencies for different sections of workflows
○ Workflows are DAG of tasks
● Schedule workflows or execute the processes by hand
○ Cron-like syntax or frequency tags
● Monitor tasks and collect/view logs

3. Airflow Principles

4. Airflow Features

5. DAGs
● DAG - Directed Acyclic Graph
○ A DAG of tasks w/ dependencies as edges
○ Individual data engineering tasks combine together to form a DAG
■ Airflow allows the definition of relationships between tasks
■ Define dependencies and run order
○ DAGs are written in python, saved as a normal .py file
● DAGs are run to a specific schedule
○ They can also be triggered manually
○ Schedule defined using CRON notation
■ Also have some tags for frequencies

6. Operators
● Operators cover some specific functionality related to tasks
○ Often built by other to cover integrations
○ Possible to supplement with custom written operators

7. Demo

8. Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037