Building Dynamic Data Pipelines in Azure Data Factory (Presented at Microsoft Ignite on November 4th, 2019)

1. @cathrinew | cathrinew.net

2. Session Abstract

3. @cathrinew
cathrinew.net

4. You are an
Azure Data Factory
developer

6. This is getting
pretty tedious…

7. There has to be
a better way!

8. Dynamic
Solutions

9. Why would you use dynamic solutions?

10. How dynamic should the solution be?

11. How dynamic should the solution be?

12. What can make a solution dynamic?
Parameters and Variables:
Pass input values and set or update values during runtime
Expressions and Functions:
Modify the content of values during runtime
Loops and Lookups:
Control logic and executions based on external configuration values

13. Parameters
and Variables

14. What are Parameters?

15. How are Parameters passed?
trigger pipeline dataset
user
activity linked service
pipeline

16. Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName

17. Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName

18. Parameters
@pipeline().parameters.
@pipeline().DataFactory
@pipeline().TriggerTime
ParameterName
and System Variables

19. What are Variables?

20. How are variables controlled?
Set Variable Append Variable

21. Variables
@variables('VariableName')
@first(variables('VariableName'))
@last(variables('VariableName'))

22. Variables
@variables('VariableName')
@first(variables('VariableName'))
@last(variables('VariableName'))

23. Expressions
and Functions

24. What are Expressions?
the @ symbol
"@toUpper(pipeline().parameters.FileName)"
"COLORS.CSV"

25. What are Functions?
String:
Date:
Collection:
Logical:
Conversion:
Math:
concat, substring, startswith, endswith …
adddays, addhours, formatDateTime …
co s, f s , s , g h …
f, , o , qu s, ss, g …
createArray, y, jso , xm , …
add, sub, mul, v, m , m x, mo , …

26. Combining Strings

27. How to combine strings?
@concat()
@{…}

28. What is String Concatenation?
@concat(
' TRUNCATE TABLE dbo. ',
pipeline().parameters.TableName
)

29. What is String Interpolation?
TRUNCATE TABLE dbo.
@{pipeline().parameters.TableName}

30. DEMO
Let's add some
parameters!

31. Loops

32. ForEach Loop Input
@item()

33. Array Items

34. Array Items

35. Array Items
colors
@item()

36. Array Items
o ch
o ch
o y
o y
o y
o y
o y
o y
@item()
parts
@item()
colors
@item()
sets

37. Object Items

38. Object Items

39. Object Items
{ "Name": "colors", "Extension": "csv" }
@item()

40. Object Items
{ "Name": "colors", "Extension": "csv" }
@item().Name

41. Object Items
o ch
o ch
o y
o y
o y
o y
o y
o y
@item().Name
parts
@item().Name
colors
@item().Name
sets

42. DEMO
Let's add a loop!

43. Lookups

44. Lookup

45. Lookup Output

46. Lookup Output

47. Lookup Output: First Row Only
{
"Name" : "colors",
"Extension" : "csv"
}
@{activity('Lookup')
.output.firstRow}

48. Lookup Output: First Row Only
{
"Name" : "colors",
"Extension" : "csv"
}
@{activity('Lookup')
.output.firstRow
.Name}

49. Lookup Output: All Rows
@{activity('Lookup')
.output.value}
[
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]

50. Lookup Output
First Row Only:
Use to pass single values to another activity
@{activity('Lookup').output.firstRow.ColumnName}
All Rows:
Use to pass entire collection to another activity
@{activity('Lookup').output.value}

51. DEMO
Let's add a lookup!

52. How do you make dynamic solutions?
1.
2. with parameters
3. solution in a loop
4. from configuration table

53. Good luck!

54. @cathrinew
cathrinew.net
hi@cathrinew.net
thank you!