The document discusses moving data pipelines into production. It describes modeling a data pipeline after a manufacturing production line by making the workflow explicit, separating responsibilities, and integrating early. It also addresses challenges of continuous integration for data-driven applications, including using synthetic data and versioning data for different branches. The conclusion emphasizes modeling the pipeline after a factory workfloor, making bottlenecks visible, considering data requirements at each step, and iterating to improve the process over time.

1. Moving Data Pipelines
Into Production
Daniel Mescheder, Real Impact Analytics

3. Raw Material
Welding
Painting
Assembly
Stress Test
Delivery

4. The Product

5. The Product
RIA
standard
Data sources
Apps
RAW
Data sources
Data
Pipeline
Backend

6. The Product
RIA
standard
Data sources
Apps
RAW
Data sources
Custom ETL Scripts
Data
Pipeline
Backend

7. The Product
Product
RIA
standard
Data sources
Apps
RAW
Data sources
Data
Pipeline
Backend

8. Requirements
Coding
QA
Business
Validation
Delivery &
Integration
Production

9. INTEGRATION
DEVELOPMENT
QA
STAGING
DEMO
DEV
QA
PO
The Factory

10. INTEGRATION
DEVELOPMENT
QA
STAGING
DEMO
DEV
QA
PO
The Factory

11. INTEGRATION
DEVELOPMENT
QA
STAGING
DEMO
DEV
QA
PO
The Factory

12. INTEGRATION
DEVELOPMENT
QA
STAGING
DEMO
DEV
QA
PO
The Factory

13. INTEGRATION
DEVELOPMENT
QA
STAGING
DEMO
DEV
QA
PO
The Factory

14. INTEGRATION
DEVELOPMENT
QA
STAGING
DEMO
DEV
QA
PO
The Factory

15. The Factory

16. The Factory
● Make Workflow Explicit
○ Highlight bottlenecks
● Separate Responsibilities
○ QA independent from development
○ PO controls the release
● Early Integration

17. Data
● Major obstacle for CI of data
driven applications
● Use of production data not
feasible
● Solution: Generator for
synthetic data
● Data is versioned: different data
for different branches
● (In the process of open sourcing
one of our tools)

18. How to get there?
Initial Investment:
Basic CI tools, create
different
environments, ...
Use it
Improve it
Retrospective

19. Conclusions
● Model your pipeline after your factory
workfloor
● Make bottlenecks visible
● Ask yourself what the data requirements
are at each step
● No one gets it right the first time - iterate!

20. Thanks*
*btw, we're hiring
@DMescheder, @RIAnalytics