Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Partition and conquer large data in PostgreSQL 10

Presentation at PGConf India 2017.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Partition and conquer large data in PostgreSQL 10

  1. 1. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 1 Partition and Conquer large data in PostgreSQL 10 • Ashutosh Bapat | 2017.02.03 @ PGConf India
  2. 2. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 2 • Manageability – Smaller data is easier to manage – Partition-wise utility commands – Easy bulk load and deletes • Query/DML Performance – Partition elimination (pruning) – Partition-wise joins, aggregate – Distribute DMLs across partitions • Data storages based on partition properties – “hot” and “cold” partitions – Foreign partitions Why to partition tables?
  3. 3. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 3 • Maintenance – No triggers, rules or constraints to maintain – Adding/deleting partition does not require more than a single command • Faster queries and DMLs – No trigger execution overhead ● DMLs are 10-20 times faster – Simpler partition bounds representation – Faster partition pruning – Partition-wise join, aggregation Why declarative partitioning? (instead of inheritance based partitioning)
  4. 4. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 4 Declarative partitioning in PostgreSQL 10
  5. 5. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 5 ● Partitioning methods: List, range, (WIP. hash) ● Partition key: Single or multiple columns, expressions ● Sub-partitioning – Partitioned partitions – Mixed sub-partitioning ● Development efforts – Several previous attempts by many people – Amit Langote proposed first patch in August 2015 – Got committed in Dec 2016, bug fixes, doc changes continue ... PostgreSQL Declarative partitioning
  6. 6. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 6 CREATE TABLE part_tab (c1 int, c2 int, …) PARTITION BY RANGE (c1); Creating partitioned table part_tab (c1 int, c2 int, ...)
  7. 7. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 7 CREATE TABLE part_tab (c1 int, c2 int, …) PARTITION BY RANGE (c1); CREATE TABLE part1 PARTITION OF part_tab FOR VALUES FROM (0) TO (100); CREATE TABLE part2 PARTITION OF part_tab FOR VALUES FROM (100) TO (200); Creating partitioned table part_tab (c1 int, c2 int, ...) Part1 0 to 100 Part2 100 to 200 Tablespaces
  8. 8. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 8 ALTER TABLE part_tab ATTACH PARTITION ext_part FOR VALUES FROM (400) to (500); ATTACH partition part_tab (c1 int, c2 int, ...) Part1 0 to 100 Part2 100 to 200 ext_part 400 to 500 ext_part (c1 int, c2 int, ...) CHECK c1 >= 400 AND c1 < 500 Bulk load data.csv
  9. 9. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 9 ALTER TABLE part_tab DETACH PARTITION ext_part; DETACH partition part_tab (c1 int, c2 int, ...) Part1 0 to 100 Part2 100 to 200 ext_part (c1 int, c2 int, ...) CHECK c1 >= 400 AND c1 < 500
  10. 10. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 10 ● Same columns as parent table ● Partition specific constraints, defaults, indexes, storage parameters ● Tablespace separate from that of parent – “hot” and “cold” partitions ● VACUUM, ANALYZE, CLUSTER can be run separately – Utilities don't block the whole table – Work where it's required Partitions are tables
  11. 11. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 11 CREATE TABLE part_tab (c1 int, c2 int, …) PARTITION BY RANGE (c1); ... CREATE FOREIGN TABLE part3 PARTITION OF part_tab FOR VALUES FROM (300) TO (400) SERVER xyz_server; CREATE FOREIGN TABLE part4 PARTITION OF part_tab FOR VALUES FROM (400) TO (500) SERVER abc_server; Foreign partitions part_tab (c1 int, c2 int, ...) Part1 0 to 100 Part2 100 to 200 Part3 300 to 400 fpart3 (c1 int, c2 int, ...) c1 >= 300 AND c1 < 400 xyz_server xyz_fdw Part4 400 to 500 fpart3 (c1 int, c2 int, ...) c1 >= 400 AND c1 < 500 abc_server abc_fdw
  12. 12. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 12 Query optimizations partition pruning partition-wise join partition-wise aggregation
  13. 13. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 13 Query optimization: partition pruning SELECT * FROM t1 WHERE c1 = 350; Partition 1 FOR VALUES FROM (0) TO (100) Partitioned table t1 (c1 int, c2 int, …) Partition 4 FOR VALUES FROM (300) TO (400) Partition 3 FOR VALUES FROM (200) TO (300) Partition 2 FOR VALUES FROM (100) TO (200) SELECT * FROM t1 WHERE c1 BETWEEN 150 AND 250; Constraintexclusion
  14. 14. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 14 Query optimization: partition-wise join Partition 1 FOR VALUES (0) TO (100) Partitioned table t1 (c1 int, ...) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200) Partition 1 FOR VALUES (0) TO (100) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200) Partition 1 FOR VALUES (0) TO (100) Partitioned table t2 (c1 int, ...) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200) t1 JOIN t2 ON t1.c1 = t2.c1 Partitioned join Partition 3 FOR VALUES (200) TO (300) Partition 1 FOR VALUES (0) TO (100) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200) Partition 3 FOR VALUES (200) TO (300) Partition 1 FOR VALUES (0) TO (100) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200)
  15. 15. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 15 Query optimization: partition-wise aggregation Partitioned table t1 (c1 int, ...) Partition 1 Partition 3 Partition 2 Partition 1 Partitioned table t2 (c1 int, ...) Partition 3 Partition 2 t1 JOIN t2 ON t1.c1 = t2.c1 Partition 1 Partition 3 Partition 2 Partition 1 Partition 3 Partition 2 Agg/Group Agg/Group Agg/Group Agg/Group
  16. 16. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 16 • Bad partitioning is worse than no partitioning • Query optimization – Partition key is the key to success – Columns in conditions – include it in the query • Storage management – Columns that decide the storage – Usually timestamp of the row – Easy to add and drop partitions • Keep an eye on current limitations – Expected to reduce with next few releases Choosing partitioning scheme
  17. 17. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 17 • We have just begun … • No triggers on partitioned table – Need to create those for each of the partitions • No SPLIT, MERGE, EXCHANGE partition • No global indexes – No primary key, unique constraints • Foreign partitions – INSERTs via parent table not supported – No ACID support for transactions involving multiple foreign servers Limitations

    Be the first to comment

    Login to see the comments

  • AmarnathGopalan

    Mar. 25, 2017
  • geoyws

    Sep. 13, 2017
  • ilhamrizqi

    Apr. 28, 2018

Presentation at PGConf India 2017.

Views

Total views

2,008

On Slideshare

0

From embeds

0

Number of embeds

12

Actions

Downloads

45

Shares

0

Comments

0

Likes

3

×