This 400 level presentation explains the U-SQL Query Execution in Azure Data Lake and provides several Performance Tuning tips: What tools are available and some best practices.

1. U-SQL Query Execution and
Performance Tuning

2. •
•
•
•
•
•

3. • Job Execution Experience and Investigations
Query Execution
Stage Graph
Dryad crash course
Job Metrics
Resource Planning
• Partitioning Analysis
Analyze the critical path
Heat Map
Critical Path
Data Skew
• Tuning / Optimizations
Data Partitioning
Partition Elimination
Predicate Pushing
Column Pruning
Some Data Hints
UDOs can be evil
INSERT optimizations
U-SQL Query Execution and Performance Tuning

5.  Automatic "in-lining"
 optimized out-of-
the-box
 Per job
parallelization
 visibility into execution
 Heatmap to identify
bottlenecks

6. Preparing
Queued
Running
Finalizing
Ended
(Succeeded, Failed, Cancelled)
New
Compiling
Queued
Scheduling
Starting
Running
Ended
What you see in the
UX
Underlying
Job State
The script is being compiled by the
Compiler Service
All jobs enter the queue.
Are there enough ADLAUs to start
the job?
If yes, then allocate those ADLAUs for
the job
The U-SQL runtime is now executing
the code on 1 or more ADLAUs or
finalizing the outputs
The job has concluded.

7. U-SQL C#
user code
C++
system code
Algebra
other files
(system files, deployed resources)
managed dll
Unmanaged dll
Input
script
Compilation output (in job folder)
Compiler & Optimizer
Files
Meta
Data
Service
Deployed to vertices

8. Some fixed amount of work
Each square is called a “vertex”
Each vertex represents a fraction of the work

9. U-SQL Query Execution
Physical plans vs. Dryad stage graph…

10. 252 Pieces of work
AVG Vertex execution time
4.3 Billion rows
Data Read & Written

11. U-SQL Query Execution
Dryad as an art form…

12. 13
U-SQL Query Execution
Redefinition of big-data…

13. 14
U-SQL Query Execution
Redefinition of big-data…

15. 16
U-SQL Performance Analysis
Analyze the critical path, heat maps, playback, and runtime metrics on every vertex…

17. 
 data may be
distributed such that
all rows that match a
certain key go to a
single vertex
 imbalanced
execution, vertex time
out.
0
5,000,000
10,000,000
15,000,000
20,000,000
25,000,000
30,000,000
35,000,000
40,000,000
California
NewYork
Illinois
Ohio
Michigan
NewJersey
Washington
Arizona
Tennessee
Maryland
Minnesota
Alabama
Louisiana
Oregon
Connecticut
Mississippi
Kansas
Nevada
Nebraska
Idaho
Maine
RhodeIsland
Delaware
Alaska
DistrictofColumbia
Wyoming
Population by State

18. 



@rows =
SELECT Gender, AGG<MyAgg>(…) AS Result
FROM @HugeInput
GROUP BY Gender;
Gender==Male Gender==Female
@HugeInput
Vertex 0 Vertex 1

19. 1 2 3 4 5 6 7 8 36
1 2 3 4 5 6 7 8
6 15 15
36

20. U-SQL Partitioning during Processing
Data Skew

21. U-SQL Partitioning
Data Skew – Recursive Reducer
// Metrics per domain
@Metric =
REDUCE @Impressions ON UrlDomain
USING new Bing.TopNReducer(count:10)
;
// …
Inherent Data Skew
[SqlUserDefinedReducer(IsRecursive = true)]
public class TopNReducer : IReducer
{
public override IEnumerable<IRow>
Reduce(IRowset input, IUpdatableRow output)
{
// Compute TOP(N) per group
// …
}
}
Recursive
• Allow multi-stage aggregation trees
• Requires same schema (input => output)
• Requires associativity:
• R(x, y) = R( R(x), R(y) )
• Default = non-recursive
• User code has to honor recursive semantics
www.bing.com
brought to a single vertex

22. U-SQL Partitioning during Processing
Partitioning – Combiner Modes
// Existing Watson hits
@DeDup =
COMBINE @Exceptions AS L WITH @WatsonBuckets AS R
ON L.AppId WITH R.AppId
USING new Windows.WatsonDedupCombiner()
;
// …
[SqlUserDefinedCombiner(Mode = CombinerMode.Right)]
public class WatsonDedupCombiner : ICombiner
{
public override IEnumerable<IRow>
Combine(IRowset left, IRowset right, IUpdatableRow output)
{
// DeDup against existing Call Stacks
// …
}
}
CombinerMode
• Allow parallelism even within a partition
public enum CombinerMode
{
Inner, /// Inner join - both row level
Left, /// Left Group Join - left row level
Right, /// Right Group Join - right row level
Full /// Full Group Join - none row level
• Default = Full Group Join
• User code has to honor row level semantics
Row Level Combiner
X
IDother columns
X
Q
Q
F
Q
X
X
Z
F
F
X
LEFT
X
ID other columns
Q
Q
F
Q
X
Z
F
F
X
RIGHT
F
Z
M1
M2
M3
M4
Enables Broadcast JOIN

23. 














24. 









Partition Scheme When to use?
HASH(keys)
DIRECT HASH Exact control of hash bucket
RANGE(keys) Keeps ranges together
ROUND ROBIN To get equal distribution (if others give skew)

26. // Unstructured Files (24 hours daily log impressions)
@Impressions =
EXTRACT ClientId int, Market string, OS string, ...
FROM @"wasb://ads@wcentralus/2015/10/30/{*}.nif"
FROM @"wasb://ads@wcentralus/2015/10/30/{Market:*}_{*}.nif"
;
// …
// Filter to by Market
@US =
SELECT * FROM @Impressions
WHERE Market == "en"
;
U-SQL Optimizations
Partition Elimination – Unstructured Files
Partition Elimination
• Even with unstructured files!
• Leverage Virtual Columns (Named)
• Avoid unnamed {*}
• WHERE predicates on named virtual columns
• That binds the PE range during compilation time
• Named virtual columns without predicate = error
• Design directories/files with PE in mind
• Design for elimination early in the tree, not in the leaves
Extracts all files in the folder
Post filter = pay I/O cost to drop most data
PE pushes this predicate to the EXTRACT
EXTRACT now only reads “en” files!
en_10.0.nif
en_8.1.nif
de_10.0.nif
jp_7.0.nif
de_8.1.nif
../2015/10/30/
…

27. // TABLE(s) - Structured Files (24 hours daily log impressions)
CREATE TABLE Impressions (Day DateTime, Market string, ClientId int, ...
INDEX IX CLUSTERED(Market, ClientId)
PARTITIONED BY
BUCKETS (Day)
HASH(Market, ClientId) INTO 100
);
DECLARE @today DateTime = DateTime.Parse("2015/10/30");
// Market = Vertical Partitioning
ALTER TABLE Impressions ADD PARTITION (@today);
// …
// Daily INSERT(s)
INSERT INTO Impressions(Market, ClientId)
PARTITION(@today)
SELECT * FROM @Q
;
// …
// Both levels are elimination (H+V)
@Impressions =
SELECT * FROM dbo.Impressions
WHERE
Market == "en"
AND Day == @today
;
U-SQL Optimizations
Partition Elimination – TABLE(s)
Partition Elimination
• Horizontal and vertical partitioning
• Horizontal is traditional within file (range, hash, robin)
• Vertical is across files (bucketing)
• Immutable file system
• Design according to your access patterns
Enumerate all partitions filtering for today
30.ss
30.1.ss
29.ss
28.ss
29.1.ss
Impressions
…
deen
jp
de
PE across files + within each file

28. @Inpressions =
SELECT * FROM
searchDM.SML.PageView(@start, @end) AS PageView
OPTION(LOWDISTINCTNESS=Query)
;
// Q1(A,B)
@Sessions =
SELECT
ClientId,
Query,
SUM(PageClicks) AS Clicks
FROM
@Impressions
GROUP BY
Query, ClientId
;
// Q2(B)
@Display =
SELECT * FROM @Sessions
INNER JOIN @Campaigns
ON @Sessions.Query == @Campaigns.Query
;
U-SQL Optimizations
Partitioning – Minimize (re)partitions
Input must be partitioned on:
(Query)
Input must be partitioned on:
(Query) or (ClientId) or (Query, ClientId)
Optimizer wants to partition only once
But Query could be skewed
Data Partitioning
• Re-Partitioning is very expensive
• Many U-SQL operators can handle multiple partitioning choices
• Optimizer bases decision upon estimations
Wrong statistics may result in worse query performance

29. // Unstructured (24 hours daily log impressions)
@Huge = EXTRACT ClientId int, ...
FROM
@"wasb://ads@wcentralus/2015/10/30/{*}.nif"
;
// Small subset (ie: ForgetMe opt out)
@Small = SELECT * FROM @Huge
WHERE Bing.ForgetMe(x,y,z)
OPTION(ROWCOUNT=500)
;
// Result (not enough info to determine simple Broadcast
join)
@Remove = SELECT * FROM Bing.Sessions
INNER JOIN @Small ON Sessions.Client ==
@Small.Client
;
U-SQL Optimizations
Partitioning - Cardinality
Broadcast JOIN right?
Broadcast is now a candidate.
Wrong statistics may result in worse query performance
Optimizer has no stats this is small...

31. // Bing impressions
@Impressions = SELECT * FROM
searchDM.SML.PageView(@start, @end) AS PageView
;
// Compute sessions
@Sessions =
REDUCE @Impressions ON Client, Market
READONLY Market
USING new Bing.SessionReducer(range : 30)
;
// Users metrics
@Metrics =
SELECT * FROM @Sessions
WHERE
Market == "en-us"
;
// …
Microsoft Confidential
U-SQL Optimizations
Predicate pushing – UDO pass-through columns

32. // Bing impressions
@Impressions = SELECT * FROM
searchDM.SML.PageView(@start, @end) AS PageView
;
// Compute page views
@Impressions =
PROCESS @Impressions
READONLY Market
PRODUCE Client, Market, Header string
USING new Bing.HtmlProcessor()
;
@Sessions =
REDUCE @Impressions ON Client, Market
READONLY Market
USING new Bing.SessionReducer(range : 30)
;
// Users metrics
@Metrics =
SELECT * FROM @Sessions
WHERE
Market == "en-us"
;
Microsoft Confidential
U-SQL Optimizations
Predicate pushing – UDO row level processors
public abstract class IProcessor : IUserDefinedOperator
{
/// <summary/>
public abstract IRow Process(IRow input, IUpdatableRow output);
}
public abstract class IReducer : IUserDefinedOperator
{
/// <summary/>
public abstract IEnumerable<IRow> Reduce(IRowset input, IUpdatableRow output);
}

33. // Bing impressions
@Impressions = SELECT Client, Market, Html FROM
searchDM.SML.PageView(@start, @end) AS PageView
;
// Compute page views
@Impressions =
PROCESS @Impressions
PRODUCE Client, Market, Header string
USING new Bing.HtmlProcessor()
;
// Users metrics
@Metrics =
SELECT * FROM @Sessions
WHERE
Market == "en-us"
&& Header.Contains("microsoft.com")
AND Header.Contains("microsoft.com")
;
U-SQL Optimizations
Predicate pushing – relational vs. C# semantics

35. // Bing impressions
@Impressions = SELECT * FROM
searchDM.SML.PageView(@start, @end) AS PageView
;
// Compute page views
@Impressions =
PROCESS @Impressions
PRODUCE *
REQUIRED ClientId, HtmlContent(Header, Footer)
USING new Bing.HtmlProcessor()
;
// Users metrics
@Metrics =
SELECT ClientId, Market, Header FROM @Sessions
WHERE
Market == "en-us"
;
U-SQL Optimizations
Column Pruning and dependencies
C H M
C H M
C H M
Column Pruning
• Minimize I/O (data shuffling)
• Minimize CPU (complex processing, html)
• Requires dependency knowledge:
• R(D*) = Input ( Output )
• Default no pruning
• User code has to honor reduced columns
A B C D E F G J KH I … M … 1000

36. • Use SELECT with UDFs instead of PROCESS
• Use User-defined Aggregators instead of REDUCE
• Hint Cardinality if you use CROSS APPLY and it does chose the
wrong plan
• Avoid ORDER BY unless needed (OUTPUT, “Top N Rows”)
• Learn to use Windowing Functions (OVER expression)
• Use SQL.MAP and SQL.ARRAY instead of C# Dictionary and array

37. Multiple INSERTs into same table
• Generates separate file per insert in physical
storage:
• Can lead to performance degradation
• Recommendations:
• Try to avoid small inserts
• Rebuild table after frequent insertions with:
ALTER TABLE T REBUILD;

38. •
•
•
•
•
•

39. Additional
Resources
 Blogs and community page:
 http://usql.io
 http://blogs.msdn.com/b/visualstudio/
 http://azure.microsoft.com/en-us/blog/topics/big-data/
 https://channel9.msdn.com/Search?term=U-SQL#ch9Search
 Documentation and articles:
 http://aka.ms/usql_reference
 https://azure.microsoft.com/en-
us/documentation/services/data-lake-analytics/
 https://msdn.microsoft.com/en-us/magazine/mt614251
 ADL forums and feedback
 http://aka.ms/adlfeedback
 https://social.msdn.microsoft.com/Forums/azure/en-
US/home?forum=AzureDataLake
 http://stackoverflow.com/questions/tagged/u-sql