Scale and Optimize Data Engineering Pipelines with Software Engineering Best Practices: Modularity and Automated Testing
•
0 likes•329 views
In rapidly changing conditions, many companies build ETL pipelines using ad-hoc strategy. Such an approach makes automated testing for data reliability almost impossible and leads to ineffective and time-consuming manual ETL monitoring.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Scale and Optimize Data Engineering Pipelines with Software Engineering Best Practices: Modularity and Automated Testing
Similar to Scale and Optimize Data Engineering Pipelines with Software Engineering Best Practices: Modularity and Automated Testing (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Scale and Optimize Data Engineering Pipelines with Software Engineering Best Practices: Modularity and Automated Testing
- 1. Scale and Optimize Data Engineering Pipelines with Software Engineering Best Practices: Modularity and Automated Testing © 2020 / Levi Strauss & Co. Qiang MENG Sr. Data Engineer, Levi Strauss & Co.
- 2. Qiang MENG • Levi Strauss & Co., Belgium • Databricks – AWS – Airflow – Spark - Dataiku • H&M Group, Sweden • Databricks – Azure – Airflow – Spark - Hive • Softbank Robotics, France • AWS – Luigi – Spark © 2020 / Levi Strauss & Co.
- 3. © 2020 / Levi Strauss & Co.
- 4. Levi’s Data, Analytics & AI © 2020 / Levi Strauss & Co. Demand Forecast Loyalty Price & Promo Recommendation 0% 10% 20% 30% 40% 50% 60% 70% Q3 Digital Sales Increase USA EU AMA NLP Allocation Computer Vision
- 5. Agenda I. Hypothesis • A Data Engineering project in Apache Airflow II. Optimization • Software Engineering best practices III. Auto-testing • Unit test, Functional test, End to End test, and Smoky test © 2020 / Levi Strauss & Co.
- 6. I. Hypothesis © 2020 / Levi Strauss & Co.
- 7. © 2020 / Levi Strauss & Co. Hypothesis Data Engineering – Cloud - Airflow - TB-level - Batch
- 8. © 2020 / Levi Strauss & Co. Hypothesis Dev Environments ▪ Local – Dev – Preprod – Prod Sample Data ▪ ETL in local / Dev Code Versioning – GIT • feature - dev - release - hotfix – master Conventions • DB, Schema, Table, Column, Pipeline Container-orchestration • Docker, K8S, … Spark clusters • Databricks, EMR, … Data Versioning • Delta Lake, … CICD
- 9. Airflow ▪ airflow • dags • operators • … ▪ utils ▪ modules ▪ docs ▪ tests ▪ others • airflow • dags • perators • … • others © 2020 / Levi Strauss & Co.
- 10. II. Optimization © 2020 / Levi Strauss & Co.
- 11. A Big Data ETL
- 12. Optimization - Airflow DAG
- 13. Optimization - Airflow DAG A simple template of tasks – Data Feeds Based © 2020 / Levi Strauss & Co.
- 14. © 2020 / Levi Strauss & Co. Optimization - Airflow Operator • Default Operators • Keep as they are • Customized Operators • Hive • Spark • Deduplication • Data Validation • Data Profiling • … Do the real work
- 15. Optimization - Modules and Utils ▪ Modules ▪ Utils
- 17. Optimization - OOP © 2020 / Levi Strauss & Co.
- 18. Optimization - Design Patten © 2020 / Levi Strauss & Co.
- 19. Optimization - Formatting © 2020 / Levi Strauss & Co.
- 20. Optimization - Docs • Docstring - Sphinx • Data Catalog – Markdown © 2020 / Levi Strauss & Co.
- 21. Docstring © 2020 / Levi Strauss & Co.
- 22. Auto Data Catalog with Markdown © 2020 / Levi Strauss & Co.
- 23. III. Auto-testing © 2020 / Levi Strauss & Co.
- 24. Auto-Testing • Unit tests • Integration tests • End-to-End tests • Smoky tests © 2020 / Levi Strauss & Co.
- 25. Unit Tests © 2020 / Levi Strauss & Co.
- 26. Integration Tests © 2020 / Levi Strauss & Co.
- 27. End-to-End Tests © 2020 / Levi Strauss & Co.
- 28. Smoke Tests © 2020 / Levi Strauss & Co.
- 29. READY FOR THE NEXT 167 YEARS © 2020 / Levi Strauss & Co.
- 30. Sign off/Thank You/Questions? © 2020 / Levi Strauss & Co. READY FOR THE NEXT 166 YEARS THANK YOU © 2020 / Levi Strauss & Co.
- 31. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions. © 2020 / Levi Strauss & Co.