Vip Model Call Girls (Delhi) Karol Bagh 9711199171✔️Body to body massage wit...
U-SQL Query Execution and Performance Basics (SQLBits 2016)
1. Michael Rys
Principal Program Manager, Big Data @ Microsoft
@MikeDoesBigData, {mrys, usql}@microsoft.com
U-SQL Query Execution and Performance
Basics
2. Simplified U-SQL Job Workflow
Job Front End
Job Scheduler Compiler Service
Job Queue
Job Manager
U-SQL Catalog
YARN
Job submission
Job execution
U-SQL Runtime Vertex execution
3. U-SQL Compilation Process
C#
C++
Algebra
Other files
(system files, deployed resources)
managed dll
Unmanaged dll
Compilation output (in job folder)
Compiler &
Optimizer
U-SQL Metadata
Service
Deployed to
Vertices
6. The Job Queue
The queue is ordered by
job priority.
Lower numbers -> higher
priority.
1 = highest.
Running jobs
When a job is at the top
of the queue, it will start
running.
Defaults:
Max Running Jobs = 3
Max Tokens per job = 20
Max Queue Size = 200
7. Priority Doesn’t Preempt Running Jobs
X has Pri=1.
X
A
B
C
X will NOT preempt running jobs. X will have to wait.
These are all running
and have very low
priority (pri=1000)
15. U-SQL Script -> Job Graph
Logical -> Physical Plan
Each square = “a vertex”
represents a fraction of the
total
Vertexes in each SuperVertex (aka
“Stage) are doing the same
operation on a different part of the
same data.
Visualized as a
“Job Graph”
17. Execution with Requested Parallelism
Requested Parallelism = 1
(reserve enough to do 1
vertex at a time)
Requested Parallelism = 4
(reserve enough to do 4
vertices at a time)
18. Notes
The next stage can
start before the
previous one has
finished
It may not be possible
to use all the reserved
parallelism during a
Stage