Data warehousing has reached a significant tipping point with changes in data sources and volumes. Traditional extract, transform, load (ETL) processes and data warehouses are evolving to incorporate streaming data, non-relational data types, and cloud-based data lakes. This provides organizations with greater flexibility to ingest, transform, and publish diverse data for analytics.
3. … data warehousing has reached the most
significant tipping point since its inception.
The biggest, possibly most elaborate data
management system in IT is changing.
– Gartner, “The State of Data Warehousing in 2012”
Data sources
5. ETL Tool
(SSIS, etc)
EDW
(SQL Svr, Teradata, etc)
Extract
Original Data
Load
Transformed
Data
Transform
BI Tools
Data Marts
Data Lake(s)
Dashboards
Apps
6. ETL Tool
(SSIS, etc)
EDW
(SQL Svr, Teradata, etc)
Extract
Original Data
Load
Transformed
Data
Transform
BI Tools
Ingest (EL)
Original Data
Data Marts
Data Lake(s)
Dashboards
Apps
7. ETL Tool
(SSIS, etc)
EDW
(SQL Svr, Teradata, etc)
Extract
Original Data
Load
Transformed
Data
Transform
BI Tools
Ingest (EL)
Original Data
Scale-out
Storage &
Compute
(HDFS, Blob Storage,
etc)
Transform & Load
Data Marts
Data Lake(s)
Dashboards
Apps
Streaming data
8. ETL Tool
(SSIS, etc)
EDW
(SQL Svr, Teradata, etc)
Extract
Original Data
Load
Transformed
Data
Transform
BI Tools
Ingest (EL)
Original Data
Scale-out
Storage &
Compute
(HDFS, Blob Storage,
etc)
Transform & Load
Data Marts
Data Lake(s)
Dashboards
Apps
Streaming data
9. BI Tools
Data Marts
Data Lake(s)
Dashboards
Apps
Data Hub
(Storage & Compute)
Data Sources
(Import From)
Move data
among Hubs
Data Hub
(Storage & Compute)
Data Sources
(Import From)
Ingest
Connect & Collect Transform & Enrich Publish
Information Production:
Ingest
Move to data mart, etc
10. BI Tools
Data Marts
Data Lake(s)
Dashboards
Apps
Data Hub
(Storage & Compute)
Data Sources
(Import From)
Data Connector:
Import from source to
Hub
Data
Connector:
Import/Export
among Hubs
Data Hub
(Storage & Compute)
Data Sources
(Import From)
Data Connector:
Import from source to
Hub
Data Connector:
Export from Hub to data
store
Connect & Collect Transform & Enrich Publish
Information Production:
• Coordination & Scheduling
• Monitoring & Mgmt
• Data Lineage
17. On Premises SQL Server Azure Blob Storage
New User View
Azure Data Factory
18. On Premises SQL Server Azure Blob Storage
AdventureWorksLTDW2014
Azure Data FactoryViewOf
New Sales
Aggregated
sales
19. ViewOf
On Premises SQL Server Azure Blob Storage
New User View
Copy “NewSales” to
Blob Storage
Cloud New Sales
Azure Data FactoryViewOf
New Sales
New User
Activity
Pipeline
20. On Premises SQL Server Azure Blob Storage
New User View
Copy New Sales to
Blob Storage
Cloud New Sales
Azure Data FactoryViewOf
Cloud New Sales
Aggregate
New Sales
AggregatedSales
HDInsight
Aggregated
Sales
Pipeline
Pipeline OnPrem SSIS
package
23. • Is my data successfully getting produced?
• Is it produced on time?
• Am I alerted quickly of failures?
• What about troubleshooting information?
• Are there any policy warnings or errors?
24.
25.
26. • Easily move data to my existing data marts for consumption by my existing BI
tools
• Azure DB
• SQL Server on premises
• Oracle
• Files
• Azure Blob content