Matthew Powers gave a presentation on optimizing Delta and Parquet data lakes. He discussed the benefits of using Delta lakes such as built-in time travel, compacting, and vacuuming capabilities. Delta lakes provide these features for free on top of Parquet files and a transaction log. Powers demonstrated how to create, compact, vacuum, partition, filter, and update Delta lakes in Spark. He showed that partitioning data significantly improves query performance by enabling data skipping and filtering at the partition level.

1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics

2. Matthew Powers, Prognos Health
Optimizing Delta / Parquet
Data Lakes
#UnifiedDataAnalytics #SparkAISummit

3. Agenda
• Why Delta?
• Delta basics and transaction log
• Compacting Delta lake
• Vacuuming old files
• Partitioning Delta lakes
• Deleting rows
• Persisting transformations in columns
3

4. About
4
MungingData
• Time travel
• Compacting
• Vacuuming
• Update columns

5. Contact me
• GitHub: MrPowers
• Email: matthewkevinpowers@gmail.com
• Delta Slack channel
• Open source hacking
5

6. What is Delta lake?
• Parquet + transaction log
• Provides awesome features for free!
6

7. Delta Lake =!= Databricks Delta
7
https://github.com/delta-io/delta/issues/49

8. #UnifiedDataAnalytics #SparkAISummit
TL;DR
• 1 GB files
• No nested directories
8

9. #UnifiedDataAnalytics #SparkAISummit 9
Delta Lake Slack says 1GB files

10. Databricks Delta autoOptimize
10

11. Why does compaction speed up
lakes?
• Parquet: files need to be listed before they are
read. Listing is expensive in object stores.
• Delta: Data is read via the transaction log.
• Easier for Spark to read partitioned lakes into
memory partitions.
11

12. Sample Data
12

13. Create Delta Data Lake
13

14. Delta Lake on Disk
14

15. _delta_log/00000000000000000000.json
15

16. Code examples
16

17. Compact Delta Data Lake
17

18. Files post-compaction
18

19. _delta_log/00000000000000000001.json
19

20. Compacting Delta lakes without breaking
downstream apps
20
https://github.com/delta-io/delta/issues/146

21. 21

22. Delta Lake Vacuum
• Files marked for removal older than the retention
period
• Default retention period is 7 days
• Not going to improve performance
22

23. Vacuum Delta Data Lake
23

24. Files post-vacuum
24

25. Optimal number of partitions
(delta)
25

26. spark-daria helps!
26

27. spark-daria on GitHub
27

28. Optimal number of partitions (parquet)
28
https://github.com/MrPowers/spark-daria/blob/master/src/main/scala/com/github/
mrpowers/spark/daria/utils/DirHelpers.scala

29. Why partition data lakes?
• Data skipping
• Massively improve query performance
• I’ve seen queries run 50-100 times faster on
partitioned lakes
29

30. Sample data
30

31. Filtering unpartitioned lake
31
== Physical Plan ==
Project [first_name#12, last_name#13, country#14]
+- Filter (((isnotnull(country#14) && isnotnull(first_name#12)) && (country#14 = Russia)) &&
StartsWith(first_name#12, M))
+- FileScan csv [first_name#12,last_name#13,country#14]
Batched: false,
Format: CSV,
Location: InMemoryFileIndex[file:/Users/powers/Documents/tmp/blog_data/people.csv],
PartitionFilters: [],
PushedFilters: [IsNotNull(country), IsNotNull(first_name), EqualTo(country,Russia),
StringStartsWith(first_name,M)],
ReadSchema: struct

32. Partitioning the data lake
32

33. Partitioned lake on disk
33

34. _delta_log/00000000000000000000.json
34

35. Filtering partitioned lake
35
== Physical Plan ==
*(1) Project [first_name#662, last_name#663, country#664]
+- *(1) Filter (isnotnull(first_name#662) && StartsWith(first_name#662, M))
+- *(1) FileScan parquet [first_name#662,last_name#663,country#664]
Batched: true,
Format: Parquet,
Location: TahoeLogFileIndex[file:/…/tmp/europe_partitioned1],
PartitionCount: 1,
PartitionFilters: [isnotnull(country#664), (country#664 = Russia)],
PushedFilters: [IsNotNull(first_name), StringStartsWith(first_name,M)],
ReadSchema: struct<first_name:string,last_name:string>

36. Comparing physical plans
36
Unpartitioned
Project [first_name#12, last_name#13, country#14]
+- Filter (((isnotnull(country#14) && isnotnull(first_name#12))
&& (country#14 = Russia)) && StartsWith(first_name#12, M))
+- FileScan csv [first_name#12,last_name#13,country#14]
Batched: false,
Format: CSV,
Location: InMemoryFileIndex[….],
PartitionFilters: [],
PushedFilters: [IsNotNull(country), IsNotNull(first_name),
EqualTo(country,Russia), StringStartsWith(first_name,M)],
ReadSchema: struct
Partitioned
Project [first_name#662, last_name#663, country#664]
+- Filter (isnotnull(first_name#662) && StartsWith(first_name#662, M))
+- FileScan parquet [first_name#662,last_name#663,country#664]
Batched: true,
Format: Parquet,
Location: TahoeLogFileIndex[file:/…/tmp/europe_partitioned1],
PartitionCount: 1,
PartitionFilters: [isnotnull(country#664), (country#664 =
Russia)],
PushedFilters: [IsNotNull(first_name),
StringStartsWith(first_name,M)],
ReadSchema: struct<first_name:string,last_name:string>

37. Directly grabbing the partitions is
faster for Parquet lakes…
37
Directly grabbing partitions was 83 times faster than relying on partition
filters for a simple query

38. Real partitioned data lake
• Updates every 3 hours
• Has 5 million files
• 15,000 files are being added every day
• Still great for a lot of queries
38

39. Creating partitioned lake (2/3)
39

40. Partitioned lake on disk (2/3)
40

41. Creating partitioned lake (3/3)
41

42. Incrementally updating
partitioned lakes
• Small file problem grows quickly
• Compaction is hard
42

43. Filtering data from a lake
43

44. We can delete rows in Delta lakes
44

45. Deleting under the hood
45

46. Append a column on the fly
46

47. Resulting DataFrame
47

48. Append a column in Delta
48

49. Delta lake downsides… not many
49

50. Contact me
• GitHub: MrPowers
• Email: matthewkevinpowers@gmail.com
• Delta Slack channel
• Open source hacking
50

51. DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT