HeroLympics Eng V03 Henk Vd Valk

SQL Server 2005 vs 2008 Integration Services World Record Performance! Henk van der Valk Workload Performance Architect Unisys ES7000 Performance Centers [email_address]

Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

About the speaker ,[object Object],[object Object],[object Object],[object Object],[object Object]

Performance Study Goals ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

What’s the big deal with the Optimized Dataflow engine in SSIS 2008? A quick overview

De SSIS Pipeline XML DB Sources Flat File Dests RAW Custom DB Flat File Custom File OLEDB Data Destination ODBC CUSTOM Raw Adapters FLATFILE Derived Column Conditional Split Aggregate Fuzzy Lookup Merge Join RAW OLEDB Data Source ODBC CUSTOM Raw Adapters FLATFILE

OS Platform memory support Fast shared memory connection for running SSIS / SQL on same system! 32bit: Each SSIS package can use 3GB RAM, (up to 20 in parallel) 64bit: Each SSIS package can use up to 2TB, practically “unlimited for now”! IA32 (32 CPU’s) IA64 (64 Cores) X64 (64 Cores) Windows Total Virtual Memory 4 GB 16 TB 16 TB Per Process Virtual Addressable Memory 2 or 3 GB 8 TB 8 TB Supported physical memory 64 GB 2 TB 2 TB

Hardware configuration ES7000 /540 16 way/16GB 32-bit 3.0 GHz Xeon MP ES7000 /420 16 way/64GB, 64-bit 1.5 GHz Itanium-2 Unisys ES7000 /one 64 Core/256GB 3.4 GHz x64

Lab Infrastructure ,[object Object]

Test approach ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Size of flat files Number of rows 10 x 21.6 GByte 10 x 100 Million

Introduction to Aggregations … Fact Table numerieke performance measurements Dimension Table Dimension Table Dimension Table Dimension Table Employee_Dim EmployeeKey EmployeeID ... Time_Dim TimeKey TheDate ... Product_Dim ProductKey ProductID ... Customer_Dim CustomerKe y CustomerID ... Location_Dim LocationKey LocationID ... Sales _ Fact TimeKey EmployeeKey ProductKey CustomerKey LocationKey Sales ...

Package details ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Aggs Col1 col2 Col3 Col4 Agg1  Agg2  Agg3  Agg4  Agg5   Agg6   Agg7   Agg8   Agg9    Agg 10   

Aggregation Package 1 st design

1 st package / base design execution

Aggregation Results 1 st package ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Katmai Data Flow Engine a new Worker thread for each execution tree

Package with Multicast Aggregating up to 1 billion lineItems ,[object Object],[object Object],[object Object],[object Object]

Base Aggregation ,[object Object],[object Object]

Optimization1: Conditional split ,[object Object],[object Object],[object Object]

Katmai Dataflow engine @ work ,[object Object],[object Object]

-Yukon- Optimization Use “Union All’s” to create parallelism ,[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],Aggregating 1 billion LineItems SQL2005 with Multicast

[object Object],[object Object],[object Object],[object Object],Aggregating 1 billion LineItems SQL2008 with Multicast

2nd package with Multicast Aggregating 1 billion lineItems ,[object Object],[object Object]

Demo Flat file input source throughput ,[object Object],[object Object],[object Object],[object Object],[object Object],5 1 15 col. 5 col. 1 col. 15

Data flow Engine threads ,[object Object],[object Object],[object Object],[object Object],[object Object]

“ World record” TPCH Data loading on the Unisys ES7000/one with Windows2008 & SQL 2008 - Bulk Inserts -

Agenda ,[object Object],[object Object],[object Object]

1 File per Filegroup? ,[object Object],[object Object],[object Object]

Initial : Maximum performance ?

64 core / 64 Bulk Inserts with –x ,[object Object],[object Object]

SQL Server with startup parameter -x

Tip: Soft Numa on 64 cores ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Tip: Sharpen data type Money type (13% improvement) ,[object Object],[object Object],[object Object]

T-SQL bulkInserts ,[object Object]

World record - TPCH Data loading With SSIS 2008 Bulk Inserts

Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

SSIS Base package – Control Flow

SSIS Base package – Data Flow ,[object Object]

IntPolicy tool / Interrupt Affinity ,[object Object]

Interrupt Affinity set for 8 network cards

Intel Pro/1000 MT ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Other flat file best practices ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

SSIS Bulk Insert package : ,[object Object],[object Object],[object Object]

IO Throughput ,[object Object],[object Object]

SSIS - IO tuning ,[object Object],[object Object]

Tip: Increase packet size ,[object Object]

[object Object],[object Object]

SQL Server 2008 startup options ,[object Object],[object Object],[object Object],[object Object],[object Object]

Result: Packages completed in less then 1800 seconds!

Windows  2008 Optimizations

Change DEP (Windows2008) ,[object Object],[object Object],[object Object]

Windows 2008 optimization ,[object Object]

Seeing Today - Securing tomorrow [email_address]

HeroLympics Eng V03 Henk Vd Valk

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to HeroLympics Eng V03 Henk Vd Valk

Similar to HeroLympics Eng V03 Henk Vd Valk (20)

Recently uploaded

Recently uploaded (20)

HeroLympics Eng V03 Henk Vd Valk