In the last decade, Apache Parquet has become the standard format to store tabular data on disk regardless of the technology stack used. This is due to its read/write performance, efficient compression technology, interoperability and especially outstanding performance with the default settings. While these default settings and access patterns already provide decent performance, by understanding the format in more detail and using recent developments, one can get much better performance, smaller files, and utilise Parquet's newer partial reading features to read even smaller subsets of a file for a given query. This talk aims to provide insight into the Parquet format and its recent development that are useful for end users' daily workflows. One only needs prior knowledge to know what a DataFrame/tabular data is.

1. Going beyond Parquet’s
default settings
Uwe Korn – QuantCo – April 2024
🔎

2. About me
• Uwe Korn
https://mastodon.social/@xhochy / @xhochy
• CTO at Data Science startup QuantCo
• Previously worked as a Data Engineer
• A lot of OSS, notably Apache {Arrow,
Parquet} and conda-forge
• PyData Südwest Co-Organizer

3. Apache Parquet
1. Data Frame storage? CSV? Why?
2. Use Parquet

4. Photo by Hansjörg Keller on Unsplash

5. Apache Parquet
1. Columnar, on-disk storage format
2. Started in 2012 by Cloudera and Twitter
3. Later, it became Apache Parquet
4. Fall 2016 brought full Python & C++ Support
5. State-of-the-art since the Hadoop era, still going strong

6. Clear bene
fi
ts
1. Columnar makes vectorized operations fast
2. E
ffi
cient encodings and compression make it small
3. Predicate-pushdown brings computation to the I/O layer
4. Language-independent and widespread; common exchange format

7. Constructing
Parquet Files

8. Parquet with pandas

9. Parquet with polars

10. Anatomy of a
fi
le

11. Anatomy of a
fi
le

12. Anatomy of a
fi
le

13. Anatomy of a
fi
le

14. Anatomy of a
fi
le

15. Photo by Gabriel Dias Pimenta on Unsplash
Tuning

16. Knobs to tune
1. Compression Algorithm
2. Compression Level
3. RowGroup size
4. Encodings

17. Data Types!?
Photo by Patrick Fore on Unsplash

18. Data Types?

19. Data Types?
• Well, actually…

20. Data Types?
• Well, actually…
• …it doesn’t save much on disk.

21. Data Types?
• Well, actually…
• …it doesn’t save much on disk.
• By choosing the optimal types (lossless cast to e.g.
fl
oat32 or uint8) on a
month of New York Taxi trips:

22. Data Types?
• Well, actually…
• …it doesn’t save much on disk.
• By choosing the optimal types (lossless cast to e.g.
fl
oat32 or uint8) on a
month of New York Taxi trips:
Saves 963 bytes 😥 of 20.6 MiB

23. Compression
Photo by cafeconcetto on Unsplash

24. Compression Algorithm

25. Compression Algorithm
• Datasets:

26. Compression Algorithm
• Datasets:
• New York Yellow Taxi Trips 2021-01

27. Compression Algorithm
• Datasets:
• New York Yellow Taxi Trips 2021-01
• New York Yellow Taxi Trips 2021-01 with a custom prediction

28. Compression Algorithm
• Datasets:
• New York Yellow Taxi Trips 2021-01
• New York Yellow Taxi Trips 2021-01 with a custom prediction
• gov.uk (House) Price Paid dataset

29. Compression Algorithm
• Datasets:
• New York Yellow Taxi Trips 2021-01
• New York Yellow Taxi Trips 2021-01 with a custom prediction
• gov.uk (House) Price Paid dataset
• COVID-19 Epidemiology

30. Compression Algorithm
• Datasets:
• New York Yellow Taxi Trips 2021-01
• New York Yellow Taxi Trips 2021-01 with a custom prediction
• gov.uk (House) Price Paid dataset
• COVID-19 Epidemiology
• Time measurements: Pick the median of
fi
ve runs

31. Compression Algorithm

32. Compression Algorithm

33. Compression Algorithm

34. Compression Level
1. For Brotli, ZStandard and GZIP, we can tune the level
2. Snappy and „none“ have a
fi
xed compression level.

35. GZIP

36. Brotli

37. ZStandard

38. ZStandard 🔬

39. ZStandard & Brotli 🔬

40. Compression

41. Compression
1. Let’s stick for now with ZStandard, as it seems a good tradeo
ff
between
speed and size.

42. Compression
1. Let’s stick for now with ZStandard, as it seems a good tradeo
ff
between
speed and size.
2. In some cases (e.g. slow network drives), it might be worth to also
considering Brotli

43. Compression
1. Let’s stick for now with ZStandard, as it seems a good tradeo
ff
between
speed and size.
2. In some cases (e.g. slow network drives), it might be worth to also
considering Brotli
• …but Brotli is relatively slow to decompress.

44. RowGroup size
1. If you plan to partially access the data, RowGroups are the common
place to
fi
lter.
2. If you want to read the whole data, less are better.
3. Compression & encoding also works better.

45. Single RowGroup

46. Encodings
1. https://parquet.apache.org/docs/
fi
le-format/data-pages/encodings/
2. We have been using RLE_DICTIONARY for all columns
3. DELTA_* encodings not implemented in pyarrow
4. Byte Stream Split a recent addition

47. Dictionary Encoding

48. RLE Encoding

49. Byte Stream Split Encoding

50. Encodings
1. Byte Stream Split sometimes is faster than dictionary encoding, but not
signi
fi
cantly
2. For high entropy columns, BSS shines

51. Hand-Crafted Delta

52. Hand-Crafted Delta
1. Let’s take the timestamps in NYC Taxi Trip
2. Sort by pickup date
3. Compute a delta column for both dates
4. 17.5% saving on the whole
fi
le.

53. Order your data
1. With our hand-crafted delta, it was worth sorting the data
2. This can help, but only worked for the Price Paid dataset in tests, there it
saved 25%, all others actually got larger

54. Summary
1. Adjusting your data types is helpful for in-memory, but have no signi
fi
cant
e
ff
ect on-disk
2. Store high-entropy
fl
oats as Byte Stream Split encoded columns
3. Check whether sorting has an e
ff
ect
4. Delta Encoding in Parquet would be useful, use handcrafted for now
5. Zstd on level 3/4 seems like a good default compression setting

55. Cost Function for compression

56. What do we get?
1. Run once with the default settings
2. Test all compression settings, but also…
1. … use hand-crafted delta.
2. … use Byte Stream Split on predictions.

57. Cost Function for compression

58. Cost Function for compression

59. Code example available at
https://github.com/xhochy/pyconde24-parquet

60. Questions?