This document discusses data quality patterns when using Azure Data Factory (ADF). It presents two modern data warehouse patterns that use ADF for orchestration: one using traditional ADF activities and another leveraging ADF mapping data flows. It also provides links to additional resources on ADF data flows, data quality patterns, expressions, performance, and connectors.
6. Modern Data Warehouse Pattern Today
Applications
Dashboards
Business/custom apps
(structured)
Logs, files, and media
(unstructured)
r
Ingest storage
Azure Storage/
Data Lake Store
Data
Loading
Azure Data
Factory
Load flat files
into data lake on a
schedule
Data processing
Read data from
files using DBFS
Orchestration
Azure Data
Factory
Extract and
transform
relational data
Azure Databricks
Serving storage
Azure SQL DW
Load processed
data into tables
optimized for
analytics
Clean and
join with
stored data
Load to SQL DW
Databases
7. Modern Data Warehouse Pattern with ADF Mapping Data Flows
Applications
Dashboards
Business/custom apps
(structured)
Logs, files, and media
(unstructured)
r Azure Storage/
Data Lake Store
Azure Data
Factory
Load files into data
lake on a schedule
Azure Data
Factory
Extract and
transform
relational data
Azure SQL DW
Load processed
data into tables
optimized for
analytics
Clean and
join disparate
data
Databases
Azure Databricks
Let’s take a look at the process through the visual user interface
Orchestration. Directs other services to execute actions as part of the transformation process.
Mapping Data Flows. Develop graphical data transformation logic at scale without writing code using Mapping Data Flows (preview).
Mapping Data Flows. Develop graphical data transformation logic at scale without writing code using Mapping Data Flows (preview).
Monitoring: Monitor pipeline and activity runs with a simple list view interface