More Related Content
Similar to Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019) (20)
More from Cathrine Wilhelmsen (20)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
- 2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Session Description
"Wait, what? Biml is not just for generating SSIS packages?"
Absolutely not! Come and see how you can use Biml (Business
Intelligence Markup Language) to save time and speed up
other Data Warehouse development tasks. You can generate
complex T-SQL statements with Biml instead of using dynamic
SQL, create test data, and even populate static dimensions.
Don't Repeat Yourself, start automating those boring, manual
tasks today!
- 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
@cathrinew
cathrinew.net
- 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…the next 50 minutes…
T-SQL Test Data Dimensions
- 6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Wait…
Can't I use
<something else>?
- 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Of course!
But Biml works well
with source metadata
- 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
And…
Biml is fun!
- 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
But... Biml? How?
- 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional Biml
- 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Crazy Fun Biml
- 12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What do you need?
- 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Preview Pane
The Power is in the...
- 14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional BimlScript
- 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional BimlScript
<# foreach (var table in RootNode.Tables) { #>
<Package Name="Load_<#=table.Name#>" />
<# } #>
<Package Name="Load_Customer" />
<Package Name="Load_Product" />
<Package Name="Load_Sales" />
- 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript to Biml
BimlExpress Preview Pane
- 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript to... ???
BimlExpress Preview Pane
- 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Crazy Fun BimlScript
<# foreach (var table in RootNode.Tables) { #>
Yay, a package! <#=table.Name#> :)
<# } #>
Yay, a package! Load_Customer :)
Yay, a package! Load_Product :)
Yay, a package! Load_Sales :)
- 20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
T-SQL from Biml
- 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList()
GetColumnAssignmentList()
GetColumnComparisonList()
- 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Column Methods
Return code fragments
Use as building blocks in custom T-SQL
Customize output by passing parameters
Filter columns by using lambda expressions
- 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Column Methods
Return code fragments
Use as building blocks in custom T-SQL
Customize output by passing parameters
Filter columns by using lambda expressions
- 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
"A lambda expression is an
anonymous function that you can
use to create delegates or
expression tree types"
- 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
"A lambda expression is an
anonymous function that you can
use to create delegates or
expression tree types"
- 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
- 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
The arrow is the lambda operator
- 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
Input parameter is on the left side
- 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
Expression is on the right side
- 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
- 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList()
Get columns for SELECT:
- 32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList()
[Col1], [Col2], [Col3]
- 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList("src")
[src].[Col1], [src].[Col2], [src].[Col3]
- 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList()
Get columns for JOIN … ON:
- 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList()
[l].[Col1] = [r].[Col1]
AND [l].[Col2] = [r].[Col2]
AND [l].[Col3] = [r].[Col3]
- 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList("!=")
[l].[Col1] != [r].[Col1]
AND [l].[Col2] != [r].[Col2]
AND [l].[Col3] != [r].[Col3]
- 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList("src", "dst")
[src].[Col1] = [dst].[Col1]
AND [src].[Col2] = [dst].[Col2]
AND [src].[Col3] = [dst].[Col3]
- 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList
("!=", "src", "dst")
[src].[Col1] != [dst].[Col1]
AND [src].[Col2] != [dst].[Col2]
AND [src].[Col3] != [dst].[Col3]
- 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList
("!=", "src", "dst", " OR ",)
[src].[Col1] != [dst].[Col1]
OR [src].[Col2] != [dst].[Col2]
OR [src].[Col3] != [dst].[Col3]
- 40. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList
GetColumnComparisonList()
GetColumnComparisonList("!=")
GetColumnComparisonList("dst", "src")
GetColumnComparisonList(c => c.IsUsedInPrimaryKey)
GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "dst", "src")
GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "!=", "dst", "src")
- 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnAssignmentList()
Get columns for UPDATE … SET:
- 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnAssignmentList()
[l].[Col1] = [r].[Col1]
,[l].[Col2] = [r].[Col2]
,[l].[Col3] = [r].[Col3]
- 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnAssignmentList("src", "dst")
[src].[Col1] = [dst].[Col1]
,[src].[Col2] = [dst].[Col2]
,[src].[Col3] = [dst].[Col3]
- 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
T-SQL from Biml
- 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml and Bogus
Create Random and Specific Test Data
- 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Bogus Project
Simple and sane fake data generator for .NET
https://github.com/bchavez/Bogus
Advanced functionality for custom objects
…or simple functionality for Biml hacks :)
- 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Address
Commerce
Company
Database
Date
Finance
Internet
Lorem
Name
Person
Phone
System
Bogus API
Supports all data types plus built-in test data, including:
- 49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Bogus Methods
<#@ assembly name="Bogus.dll" #>
<#@ import namespace="Bogus" #>
<# var f = new Faker(); #>
<# Person p = new Person(); #>
INSERT INTO dbo.Employee(FirstName, LastName, Birthday) VALUES
(
'<#=p.FirstName#>',
'<#=p.LastName#>',
'<#=p.DateOfBirth.ToString("yyyyMMdd")#>'
)
- 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BOgus Installation
Install via Nuget:
Install-Package Bogus
Or download latest release .zip:
https://github.com/bchavez/Bogus/releases
- 51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Test Data
with Bogus
- 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Static Dimensions
- 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Static Dimensions
SCD Type 0: Not changing
• Unknown dimension members
• Date dimension
• Code tables
- 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Static Dimension Creation
1. Create table definition
2. Add static source rows
3. Generate INSERT statements
- 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Sources
Part of <Table> objects
Defines rows to be inserted when table is created
- 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Source
<Table Name="Table" SchemaName="Database.Schema">
<Columns>
<Column Name="Column1" DataType="Int32" />
<Column Name="Column2" DataType="String" Length="10" />
</Columns>
<Sources>
<StaticSource Name="TableRows">
...
</StaticSource>
</Sources>
</Table>
- 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Source
<StaticSource Name="TableRows">
<Rows>
<Row>
<ColumnValues>
<ColumnValue ColumnName="Col1" Value="-1" />
<ColumnValue ColumnName="Col2" Value="'N/A'" />
</ColumnValues>
</Row>
</Rows>
</StaticSource>
- 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Source
<# if (table.Sources.Any()) { #>
<#=TableToPackageLowerer.GetStaticSourceInsertStatements(
table.SchemaQualifiedName,
table.HasIdentity,
(AstTableStaticSourceNode)table.Sources.FirstOrDefault()
)#>
<# } #>
- 59. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Static Dimensions
- 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…the Past 50 minutes…
T-SQL Test Data Dimensions
- 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Is this useful?
- 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What other
ideas do you
have?
- 64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
Biml resources and demo files:
cathrinew.net/biml