Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Data Toboggan 2021)

•

1 like•923 views

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Presented at Data Toboggan on January 30th, 2021)

Data & Analytics

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipelines & Data Flows:
Introduction to Data Integration
in Azure Synapse Analytics
Cathrine Wilhelmsen
Data Toboggan · January 30th, 2021

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Session Abstract
Do you regularly need to get data for your projects?
Yep! 🙋‍♀️
Data is at the core of every Business Intelligence, Data Science, and Machine Learning project.
You need data to understand what has happened in the past, to predict what may happen in
the future, to discover patterns and anomalies, and to gain the insight necessary for making
faster and better decisions.
But before you can do any of those things, you need to ingest, store, transform, integrate, and
prepare your data. Guess what? You can do all of those things in Azure Synapse Analytics -
without having to write any code!
In this session, we will cover the fundamentals of data integration in Azure Synapse Analytics.
First, we will go through what Pipelines and Data Flows are. Then, we will quickly build a
solution for ingesting and transforming data. Finally, we will look at how to orchestrate and
schedule our pipelines, and how to monitor our solution once it has been deployed.

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Relational Data
Data Warehouse
Big Data
Data Lake

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Integration
Data

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Integration in Azure Synapse
Code-Heavy
Scripts, Notebooks
Low-Code
Pipelines, Data Flows

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Integration in Azure Synapse
Copy Data Transform Data

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Pipelines?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Activities?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Datasets?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Linked Services?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Data Flows?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Triggers?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Code-Heavy
Scripts, Notebooks
Low-Code
Pipelines, Data Flows

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
cathrinew.net/adf

What's hot

Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)

Cathrine Wilhelmsen

Data Quality Patterns in the Cloud with Azure Data Factory

Mark Kromer

Data Lakehouse, Data Mesh, and Data Fabric (r1)

James Serra

Embarking on building a modern data warehouse in the cloud can be an overwhelming experience due to the sheer number of products that can be used, especially when the use cases for many products overlap others. In this talk I will cover the use cases of many of the Microsoft products that you can use when building a modern data warehouse, broken down into four areas: ingest, store, prep, and model & serve. It’s a complicated story that I will try to simplify, giving blunt opinions of when to use what products and the pros/cons of each.

Building a modern data warehouse

James Serra

Microsoft Data Integration Pipelines: Azure Data Factory and SSIS

Mark Kromer

Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.

Azure Synapse Analytics Overview (r1)

James Serra

Azure Synapse Analytics

WinWire Technologies Inc

Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)

Cathrine Wilhelmsen

Intro to Azure Data Factory v1

Eric Bragas

I published a 1-hour youtube video that covers all the essential topics that are there to know about the Microsoft Azure Data Fundamentals DP 900 exam. I made sure to only include relevant exam-related topics and not to bombard you with a lot of irrelevant details at the same time, I wanted to cover the basics of each topic with a demo wherever necessary. I also wanted to validate the content of my video hence, I gave the exam before publishing the video and got an easy 900 marks with just the content I published in this video. If you plan to give this certification exam or are interested in learning Azure data fundamentals DP 900 concepts, feel free to check out this video. https://youtu.be/jopyoCgQjkM Please watch the video till the end as I have included important tips and pointers to the exam in each of the topics which would help you with lots of questions in the Microsoft Azure data fundamentals DP 900 exam. This video is sufficient for you to pass the exam. Good luck!

Azure Data Fundamentals DP 900 Full Course

Piyush sachdeva

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Cathrine Wilhelmsen

Modern Data Architecture

Alexey Grishchenko

The data lake has become extremely popular, but there is still confusion on how it should be used. In this presentation I will cover common big data architectures that use the data lake, the characteristics and benefits of a data lake, and how it works in conjunction with a relational data warehouse. Then I’ll go into details on using Azure Data Lake Store Gen2 as your data lake, and various typical use cases of the data lake. As a bonus I’ll talk about how to organize a data lake and discuss the various products that can be used in a modern data warehouse.

Data Lake Overview

James Serra

TechEvent Databricks on Azure

Trivadis

Azure Synapse Analytics Overview (r2)

James Serra

Understanding Azure Data Factory: The What, When, and Why (NIC 2020)

Cathrine Wilhelmsen

Snowflake Architecture.pptx

chennakesava44

Microsoft Azure Data Factory Hands-On Lab Overview Slides

Mark Kromer

Traditional data warehouse vs data lake

BHASKAR CHAUDHURY

Azure Data Factory

HARIHARAN R

What's hot (20)

Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)

Data Quality Patterns in the Cloud with Azure Data Factory

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Building a modern data warehouse

Microsoft Data Integration Pipelines: Azure Data Factory and SSIS

Azure Synapse Analytics Overview (r1)

Azure Synapse Analytics

Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)

Intro to Azure Data Factory v1

Azure Data Fundamentals DP 900 Full Course

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Modern Data Architecture

Data Lake Overview

TechEvent Databricks on Azure

Azure Synapse Analytics Overview (r2)

Understanding Azure Data Factory: The What, When, and Why (NIC 2020)

Snowflake Architecture.pptx

Microsoft Azure Data Factory Hands-On Lab Overview Slides

Traditional data warehouse vs data lake

Azure Data Factory

Similar to Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Data Toboggan 2021)

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

Cathrine Wilhelmsen

AWS DC Summit - Data Led Migration

Sandy Carter

Data modernization is key to unlocking the full potential of your IT investments, both on premises and in the cloud. Enterprises and organizations of all sizes rely on their data to power advanced analytics, machine learning, and artificial intelligence. Yet the path to modernizing legacy data systems for the cloud is full of pitfalls that cost time, money, and resources. These issues include high hardware and staffing costs, difficulty moving data and analytical processes to cloud environments, and inadequate support for real-time use cases. These issues delay delivery timelines and increase costs, impacting the return on investment for new, cutting-edge applications. Watch this webinar in which James Kobielus, TDWI senior research director for data management, explores how enterprises are modernizing their mainframe data and application infrastructures in the cloud to sustain innovation and drive efficiencies. Kobielus will engage John de Saint Phalle, senior product manager at Precisely, in a discussion that addresses the following key questions: When should enterprises consider migrating and replicating all their data assets to modern public clouds vs. retaining some on-premises in hybrid deployments?How should enterprises modernize their legacy data and application infrastructures to unlock innovation and value in the age of cloud computing?What are the key investments that enterprises should make to modernize their data pipelines to deliver better AI/ML applications in the cloud?What is the optimal data engineering workflow for building, testing, and operationalizing high-quality modern AI/ML applications in the cloud?What value does real-time replication play in migrating data and applications to modern cloud data architectures?What challenges do enterprises face in ensuring and maintaining the integrity, fitness, and quality of the data that they migrate to modern clouds?What tools and methodologies should enterprise application developers use to refactor and transform legacy data applications that have migrated to modern clouds

Data and Application Modernization in the Age of the Cloud

redmondpulver

It is a fascinating, explosive time for enterprise analytics. It is from the position of analytics leadership that the mission will be executed and company leadership will emerge. The data professional is absolutely sitting on the performance of the company in this information economy and has an obligation to demonstrate the possibilities and originate the architecture, data, and projects that will deliver analytics. After all, no matter what business you’re in, you’re in the business of analytics. The coming years will be full of big changes in enterprise analytics and Data Architecture. William will kick off the fourth year of the Advanced Analytics series with a discussion of the trends winning organizations should build into their plans, expectations, vision, and awareness now.

2022 Trends in Enterprise Analytics

DATAVERSITY

Creating Visual Transformations in Azure Data Factory (dataMinds Connect)

Cathrine Wilhelmsen

Analytics in a Day Virtual Workshop

CCG

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)

Cathrine Wilhelmsen

Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)

Cathrine Wilhelmsen

Big data represents a significant paradigm shift in enterprise technology. Big data radically changes the nature of the data management profession as it introduces new concerns about the volume, velocity and variety of corporate data. Apache Hadoop is the open source defacto standard for implementing big data solutions on the Java platform. Hadoop consists of its kernel, MapReduce, and the Hadoop Distributed Filesystem (HDFS). A challenging task is to send all data to Hadoop for processing and storage (and then get it back to your application later), because in practice data comes from many different applications (SAP, Salesforce, Siebel, etc.) and databases (File, SQL, NoSQL), uses different technologies and concepts for communication (e.g. HTTP, FTP, RMI, JMS), and consists of different data formats using CSV, XML, binary data, or other alternatives. This session shows different open source frameworks and products to solve this challenging task. Learn how to use every thinkable data with Hadoop – without plenty of complex or redundant boilerplate code.

"Big Data beyond Apache Hadoop - How to Integrate ALL your Data" - JavaOne 2013

Kai Wähner

Achieving agility in data and analytics is hard. It’s no secret that most data organizations struggle to deliver the on-demand data products that their business customers demand. Recently, there has been much hype around new design patterns that promise to deliver this much sought-after agility. In this webinar, Chris Bergh, CEO and Head Chef of DataKitchen will cut through the noise and describe several elegant and effective data architecture design patterns that deliver low errors, rapid development, and high levels of collaboration. He’ll cover: • DataOps, Data Mesh, Functional Design, and Hub & Spoke design patterns; • Where Data Fabric fits into your architecture; • How different patterns can work together to maximize agility; and • How a DataOps platform serves as the foundational superstructure for your agile architecture.

DataOps - The Foundation for Your Agile Data Architecture

DATAVERSITY

JAZOON'13 - Kai Waehner - Hadoop Integration

jazoon13

WJAX 2013 Slides online: Big Data beyond Apache Hadoop - How to integrate ALL...

Kai Wähner

Organizations with on-premises Hadoop infrastructure are bogged down by system complexity, unscalable infrastructure, and the increasing burden on DevOps to manage legacy architectures. Costs and resource utilization continue to go up while innovation has flatlined. In this session, you will learn why, now more than ever, enterprises are looking for cloud alternatives to Hadoop and are migrating off of the architecture in large numbers. You will also learn how elastic compute models’ benefits help one customer scale their analytics and AI workloads and best practices from their experience on a successful migration of their data and workloads to the cloud.

Modernizing to a Cloud Data Architecture

Databricks

Modernizing your infrastructure can get complicated really fast. The keys to success involve breaking down data silos and moving data to the cloud in real time. But building data pipelines to mobilize your data in the cloud can be time consuming. You need solutions that decrease bandwidth, ensure data consistency, and enable data migration and replication in real-time; solutions that help you build data pipelines in hours, not days. Watch this on-demand webinar to learn about the trends and pitfalls related to modernizing your infrastructure to cloud, how the pace of on-prem data growth demands accelerating data streaming to analytics platforms, and why mobilizing your data for the cloud improves business outcomes.

Modernize your Infrastructure and Mobilize Your Data

Precisely

Analytics in a Day Ft. Synapse Virtual Workshop

CCG

Watch full webinar here: https://bit.ly/3FF1ubd In the recent Building the Unified Data Warehouse and Data Lake report by leading industry analysts TDWI, we have discovered 64% of organizations stated the objective for a unified Data Warehouse and Data Lakes is to get more business value and 84% of organizations polled felt that a unified approach to Data Warehouses and Data Lakes was either extremely or moderately important. In this session, you will learn how your organization can apply a logical data fabric and the associated technologies of machine learning, artificial intelligence, and data virtualization can reduce time to value. Hence, increasing the overall business value of your data assets. KEY TAKEAWAYS: - How a Logical Data Fabric is the right approach to assist organizations to unify their data. - The advanced features of a Logical Data Fabric that assist with the democratization of data, providing an agile and governed approach to business analytics and data science. - How a Logical Data Fabric with Data Virtualization enhances your legacy data integration landscape to simplify data access and encourage self-service.

Building a Logical Data Fabric using Data Virtualization (ASEAN)

Denodo

If there were a buzzword of the hour, it would certainly be "data mesh"! This new architectural paradigm unlocks analytic data at scale and enables rapid access to an ever-growing number of distributed domain datasets for various usage scenarios. As such, the data mesh addresses the most common weaknesses of the traditional centralized data lake or data platform architecture. And the heart of a data mesh infrastructure must be real-time, decoupled, reliable, and scalable. This presentation explores how Apache Kafka, as an open and scalable decentralized real-time platform, can be the basis of a data mesh infrastructure and - complemented by many other data platforms like a data warehouse, data lake, and lakehouse - solve real business problems. There is no silver bullet or single technology/product/cloud service for implementing a data mesh. The key outcome of a data mesh architecture is the ability to build data products; with the right tool for the job. A good data mesh combines data streaming technology like Apache Kafka or Confluent Cloud with cloud-native data warehouse and data lake architectures from Snowflake, Databricks, Google BigQuery, et al.

The Heart of the Data Mesh Beats in Real-Time with Apache Kafka

Kai Wähner

The Briefing Room with Dean Abbott and Tableau Software Live Webcast July 23, 2013 http://www.insideanalysis.com Today’s desire for analytics extends well beyond the traditional domain of Business Intelligence. That’s partly because business users are realizing the value of mixing and matching all kinds of data, from all kinds of sources. One emerging market driver is Cloud-based data, and the desire companies have to analyze this data cohesively with their on-premise data sets. Register for this episode of The Briefing Room to learn from Analyst Dean Abbott, who will explain how the ability to access data in the cloud can play a critical role for generating business value from analytics. He’ll be briefed by Ellie Fields of Tableau Software who will tout Tableau’s latest release, which includes native connectors to cloud-based applications like Salesforce.com, Amazon Redshift, Google Analytics and BigQuery. She’ll also demonstrate how Tableau can combine cloud data with other data sources, including spreadsheets, databases, cubes and even Big Data.

Bridging the Gap: Analyzing Data in and Below the Cloud

Inside Analysis

Have you ever wondered what your environment would look like in the Cloud? How much would it cost? What would be the migration path? With Azure Migrate you and start to answer those questions. The tool can be run in your environment to help understand the tight interdependencies between applications and workloads that exist, then give you an analysis of what your Azure costs would look like. From this session you will walk away with a good idea of how to start utilising Azure Migrate in your organisation.

Start your datacentre transformation journey with azure migrate

Sarah Lean

Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)

Cathrine Wilhelmsen

Similar to Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Data Toboggan 2021) (20)

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

AWS DC Summit - Data Led Migration

Data and Application Modernization in the Age of the Cloud

2022 Trends in Enterprise Analytics

Creating Visual Transformations in Azure Data Factory (dataMinds Connect)

Analytics in a Day Virtual Workshop

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)

Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)

"Big Data beyond Apache Hadoop - How to Integrate ALL your Data" - JavaOne 2013

DataOps - The Foundation for Your Agile Data Architecture

JAZOON'13 - Kai Waehner - Hadoop Integration

WJAX 2013 Slides online: Big Data beyond Apache Hadoop - How to integrate ALL...

Modernizing to a Cloud Data Architecture

Modernize your Infrastructure and Mobilize Your Data

Analytics in a Day Ft. Synapse Virtual Workshop

Building a Logical Data Fabric using Data Virtualization (ASEAN)

The Heart of the Data Mesh Beats in Real-Time with Apache Kafka

Bridging the Gap: Analyzing Data in and Below the Cloud

Start your datacentre transformation journey with azure migrate

Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)

More from Cathrine Wilhelmsen

Data Factory in Microsoft Fabric (MsBIP #82)

Cathrine Wilhelmsen

Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...

Cathrine Wilhelmsen

Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)

Cathrine Wilhelmsen

Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...

Cathrine Wilhelmsen

Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)

Cathrine Wilhelmsen

The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)

Cathrine Wilhelmsen

Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...

Cathrine Wilhelmsen

Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...

Cathrine Wilhelmsen

Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)

Cathrine Wilhelmsen

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

Cathrine Wilhelmsen

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...

Cathrine Wilhelmsen

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (Presented at SQLBits on March 18th, 2023) We all experience stress in our lives. When the stress is time-limited and manageable, it can be positive and productive. This kind of stress can help you get things done and lead to personal growth. However, when the stress stretches out over longer periods of time and we are unable to manage it, it can be negative and debilitating. This kind of stress can affect your mental health as well as your physical health, and increase the risk of depression and burnout. The tricky part is that both depression and burnout can hit you hard without the warning signs you might recognize from stress. Where stress barges through your door and yells "hey, it's me!", depression and burnout can silently sneak in and gradually make adjustments until one day you turn around and see them smiling while realizing that you no longer recognize your house. I know, because I've dealt with both. And when I thought I had kicked them out, they both came back for new visits. I don't have the Answers™️ or Solutions™️ to how to keep them away forever. But in hindsight, there were plenty of warning signs I missed, ignored, or was oblivious to at the time. In this deeply personal session, I will share my story of dealing with both depression and burnout. What were the warning signs? Why did I miss them? Could I have done something differently? And most importantly, what can I - and you - do to help ourselves or our loved ones if we notice that something is not quite right?

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...

Cathrine Wilhelmsen

"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced World (Presented at SQLBits on March 17th, 2023) Do you sometimes think the world is moving so fast that you're struggling to keep up? Does it make you feel a little uncomfortable? Awesome! That means that you have ambitions. You want to learn new things, take that next step in your career, achieve your goals. You can do anything if you set your mind to it. It just might not be easy. All growth requires some discomfort. You need to manage and balance that discomfort, find a way to push yourself a little bit every day without feeling overwhelmed. In a fast-paced world, you need to know how to break down your goals into smaller chunks, how to prioritize, and how to optimize your learning. Are you ready to turn your "I can't keep up" into "I can't believe I did all of that in just one year"?

"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...

Cathrine Wilhelmsen

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing Startup (Presented at SQLBits on March 11th, 2022) What happens when you mix one rapidly-changing startup, one data analyst, one data engineer, and one hypothesis that Azure Synapse Analytics could be the right tool of choice for gaining business insights? We had no idea, but we gave it a go! Our ambition was to think big, start small, and act fast – to deliver business value early and often. Did we succeed? Join us for an honest conversation about why we decided to implement Azure Synapse Analytics alongside Power BI, how we got started, which areas we completely messed up at first, what our current solution looks like, the lessons learned along the way, and the things we would have done differently if we could start all over again.

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...

Cathrine Wilhelmsen

6 Tips for Building Confidence as a Public Speaker (Presented at SQLBits on March 10th, 2022) Do you feel nervous about getting on stage to deliver a presentation? That was me a few years ago. Palms sweating. Hands shaking. Voice trembling. I could barely breathe and talked at what felt like a thousand words per second. Now, public speaking is one of my favorite hobbies. Sometimes, I even plan my vacations around events! What changed? There are no shortcuts to building confidence as a public speaker. However, there are many things you can do to make the journey a little easier for yourself. In this session, I share the top tips I have learned over the years. All it takes is a little preparation and practice. You can do this!

6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)

Cathrine Wilhelmsen

Azure Data Factory for the SSIS Developer (SentryOne Webinar)

Cathrine Wilhelmsen

Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)

Cathrine Wilhelmsen

Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...

Cathrine Wilhelmsen

Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019)

Cathrine Wilhelmsen

Uhms and Bunny Hands: Tips for Improving Your Presentation Skills (DataGrille...

Cathrine Wilhelmsen

More from Cathrine Wilhelmsen (20)