Airbnb's ML Data Management Framework Zipline
•
3 likes•570 views
Zipline is Airbnb's machine learning data management framework. It handles feature engineering, discovering and accessing data sources, generating training sets, and monitoring data quality. Zipline includes a feature store, training set generation, and clients to access features and training data. It uses both batch and streaming processing with Apache Spark and Flink to handle training set generation and updates from mutation streams in near real-time. Airbnb plans to open source Zipline in Q3 2018 as part of their end-to-end machine learning platform called Bighead.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Airbnb's ML Data Management Framework Zipline
Similar to Airbnb's ML Data Management Framework Zipline (20)
More from Karthik Murugesan
More from Karthik Murugesan (20)
Recently uploaded
Recently uploaded (20)
Airbnb's ML Data Management Framework Zipline
- 1. Nikhil Simha, Airbnb Andrew Hoh, Airbnb Zipline – Airbnb’s ML Data Management Framework #ML3SAIS
- 2. Feature Engineering 2#ML3SAIS Idea Discover Data Training Set Training & Evaluation Airbnb Zipline
- 3. Discovering data 3#ML3SAIS • Where is it? • Is it good? • Will it continue to be good? • Has it already been done before?
- 4. Training set 4#ML3SAIS • What processing engine to use? • Backfilling • Point-in-time correct windows
- 5. Production 5#ML3SAIS Training Set Generation Model Training Push model to prediction service Online Features Model monitoring Data quality monitoring
- 6. What is Zipline? • Part of Bighead – Airbnb’s e2e ml platform • Training Sets • Feature Store • Data quality 6#ML3SAIS
- 7. Concepts • Data Source • Feature-set • Training-set • Client 7#ML3SAIS
- 8. Feature-Set • Source queries – Hive tables – Event streams – Mutation streams • Primary Keys • Time Stamp 8#ML3SAIS
- 9. Feature-Set – Sources • Multiple sources – Backfills – Migrations 9#ML3SAIS
- 11. 11#ML3SAIS The picture can't be displayed. Data Quality
- 12. Training set • Driver query – primary keys – timestamps – Point-in-time correct 12#ML3SAIS
- 13. Training set – example 13#ML3SAIS
- 14. Training set – example 14#ML3SAIS
- 17. Online scoring • Backfills • Low latency • DB Mutations – Batch correction 17#ML3SAIS
- 18. Lambda • Batch – Spark • Streaming – Flink • GDPR 18#ML3SAIS User Conf Batch Streaming Zipline Client User App *Daily *Continuous KV Store
- 21. • Across batch and streaming • Fixed length sliding windows • Raw Events at the tail and head Aggregations 21#ML3SAIS 7 day sliding window
- 23. Client 23#ML3SAIS • Aggregation • Logic to handle availability • Java – The API is (List of Feature Names) => Map of feature values – as objects.
- 24. Mutations • Organizationally easy • Consistent • Not flexible • More complex – Inverse 24#ML3SAIS
- 25. Mutations - Aggregation 25#ML3SAIS • Monoids – Associative: (a + b) + c = a + (b + c) – Distribute aggregation – Sum, Avg, Count etc.. • Group – Invertible – Min, Max, Median, Ntile – Not possible without memory – Compromise
- 26. Mutations - Aggregation 26#ML3SAIS • Deletes = Inverse(before) • Updates = Inverse(before) + after
- 28. When? • Plan to open source in Q3 2018. • Will be part of Bighead – Talk tomorrow. 28#ML3SAIS