http://coscup.org/2015/zh-tw/program/#/

1. PostgreSQL 9.4, 9.5 and
Beyond
JSON, Analytics, and More
Uptime Technologies
Satoshi Nagayasu
@snaga
COSCUP 2015

2. Who I Am
• Satoshi Nagayasu
– Database enthusiast. DBA and Data Steward.
– Traveling in Asia: Hong Kong, Shenzhen, Beijing, Singapore,
Taipei
• Uptime Technologies
– Co-founder
– Consulting services around Database and Platform
Technologies.
• PostgreSQL
– pgstatindex, pageinspect, xlogdump
– PostgresForest, Postgres-XC (cluster technology)
– Organizing Japanese Users Group.

3. Thanks to...
• Magnus Hagander
• Michael Paquier
• Toshi Harada
• Noriyoshi Shinoda
• ... and many pg guys!

4. Agenda
• 9.4 & 9.5 Overview
• NoSQL (JSON and GIN Index)
• Analytics (Aggregation, Mat.View & BRIN)
• SQL (UPSERT)
• Security (Row Level Security)
• Replication and Beyond (Logical Decoding)
• Administration (ALTER SYSTEM)
• Infrastructure (For Parallelization)
• Beyond 9.5

5. 9.4 & 9.5 Overview

6. 9.4 Status
• The first official release.
– 9.4 released on December 18th, 2014.
• The latest stable release
– 9.4.4 released on June 12th, 2015.

7. 9.5 Status
• Current Status
– 9.5 Alpha 2 released on August 5th, 2015
• The first Beta (or another Alpha) will be
coming in September.
• The final release will be coming on late
this year.
http://www.postgresql.org/about/news/1604/

8. Statistics
• 9.4.4 - compared to 9.3.9
– 3,910 files changed.
– 51,724 insertions (+) in *.c, *.h files
– 16,387 deletions (-)
• 9.5alpha2 - compared to 9.4.4
– 4,246 files changed.
– 91,138 insertions (+)
– 22,393 deletions (-)

9. 9.4 Overview - Changes
Lots of changes!!

10. Categories of Enhancements
• NoSQL (JSON and GIN Index)
• Analytics (Aggregation, Mat.View & BRIN)
• SQL (UPSERT)
• Security (Row Level Security)
• Replication+ (Logical Decoding)
• Administration (ALTER SYSTEM)
• Basic Infrastructure (Parallelization)

11. NoSQL
(JSON and GIN Index)

12. NoSQL - JSONB
• JSON (9.3) vs. JSONB (9.4)

13. NoSQL - JSONB
• “Binary JSON”
– Different from JSON, a text representation
– Faster for searching
• With JSONB...
– No duplicated keys allowed. Last wins.
– Key order not preserved.
– Can take advantages of GIN Index.

14. NoSQL - JSONB
Operator Description
9.4
-> Get an element by key as a JSON object
->> Get an element by key as a text object
#> Get an element by path as a JSON object
#>> Get an element by path as a text object
<@, @> Evaluate whether a JSON object contains a key/value pair
? Evaluate whether a JSON object contains a key or a value
?| Evaluate whether a JSON object contains ANY of keys or values
?& Evaluate whether a JSON object contains ALL of keys or values
9.5
|| Insert or Update an element to a JSON object
- Delete an element by key from a JSON object
#- Delete an element by path from a JSON object
http://www.postgresql.org/docs/9.5/static/functions-json.html

15. NoSQL - GIN Index
• JSON+btree vs. JSONB+GIN
– Btree indexes vs. GIN index
http://www.slideshare.net/toshiharada/jpug-studyjsonbdatatype20141011-40103981
Table Index Size Comparison

16. Analytics
(Aggregation & Materialized View)

17. Analytics - Aggregation
• FILTER replaces CASE WHEN.

18. Analytics - Aggregation
• New Aggregate Functions (New in 9.4)
– percentile_cont()
– percentile_disc()
– mode()
– rank()
– dense_rank()
– percent_rank()
– cume_dist()

19. Analytics - Aggregation
• Ordered-set aggregates
– mode(), most common value in a subset

20. Analytics - Aggregation
• Ordered-set aggregates
– rank(), rank of a value in a subset

21. Analytics - Aggregation
• New in 9.5
–ROLLUP()
–CUBE()
–GROUPING SETS()

22. Analytics - ROLLUP
• Calculates total/subtotal values

23. Analytics - CUBE
• Calculates for all combinations of the
specified columns

24. Analytics – GROUPING SETS
• Runs multiple GROUP BY queries at once
Two GROUP BYs
at once.

25. Analytics – Materialized
Views
• REFRESH MATERIALIZED VIEW
CONCURRENTLY myview
• Refreshing a MV concurrently (in
background) without exclusive lock.
• Usability and availability improved.

26. Analytics - BRIN Index
• Block Range INdex (New in 9.5)
– Holds "summary“ data, instead of raw data.
– Reduces index size tremendously.
– Also reduces creation/maintenance cost.
– Needs extra tuple fetch to get the exact record.
0
50,000
100,000
150,000
200,000
250,000
300,000
Btree BRIN
Elapsed time (ms)
Index Creation
0
50,000
100,000
150,000
200,000
250,000
300,000
Btree BRIN
Number of Blocks
Index Size
0
2
4
6
8
10
12
14
16
18
Btree BRIN
Elapsed time (ms)
Select 1 record
https://gist.github.com/snaga/82173bd49749ccf0fa6c

27. Analytics - BRIN Index
• Structure of BRIN Index
Table File
Block Range 1 (128 Blocks)
Block Range 2
Block Range 3
Block
Range
Min. Value Max. Value
1 1992-01-02 1992-01-28
2 1992-01-27 1992-02-08
3 1992-02-08 1992-02-16
… … …
Holds min/max values
for “Block Ranges”,
128 blocks each
(by default).
(in case a date
column)

28. Analytics - TABLESAMPLE
• Allows user to specify random BERNOULLI
sampling or block level SYSTEM sampling
– Would improve SELECT query performance
• Need to specify
– Sampling method (SYSTEM | BERNOULLI)
– Fraction of the table (in a percentage)
• Limitation
– Currently accepts only on regular tables and materialized
views
http://www.postgresql.org/docs/9.5/static/sql-select.html

29. Analytics - TABLESAMPLE
• Calculating the average of total price.
– With/without TABLESAMPLE

30. Analytics - TABLESAMPLE
Without TABLESAMPLE
Cost: 44076
With SYSTEM Sampl.
Cost: 1199
With BERNOULLI Sampl.
Cost: 25513

31. UPSERT

32. INSERT, or UPDATE?
• “duplicate key violation” is one of the
bothersome things in database programming

33. INSERT … ON CONFLICT …
• Now, you can INSERT or UPDATE in one
statement with “ON CONFLICT”.

34. INSERT … ON CONFLICT …
INSERT INTO nation VALUES (
12, ‐‐ n_nationkey
'JAPAN', ‐‐ n_name
2, ‐‐ n_regionkey
'Japan (Japanese: 日本 … in East Asia.' ‐‐ n_comment
)
ON CONFLICT (n_nationkey)
DO UPDATE SET n_comment = EXCLUDED.n_comment;
http://www.postgresql.org/docs/9.5/static/sql-insert.html
• This query updates n_comment column when
INSERT conflicts on n_nationkey column.

35. Row Level Security

36. Row Level Security
• Row Level Security (RLS)
– Allows users to define access policy to
determine which rows in the table should be
returned.
– Disabled by default.
– CREATE POLICY, ALTER POLICY, DROP
POLICY
• Limitation
– Not applicable to the system catalog
http://www.postgresql.org/docs/9.5/static/ddl-rowsecurity.html

37. Row Level Security
• Multiple rows (user records) in the table.
Define a policy to filter
out with user name

38. Row Level Security
• Each user can see only own record.
“user01” can see only
“user01” record
“user02” can see only
“user02” record

39. Replication and Beyond
(Logical Decoding)

40. Replication and Beyond –
Logical Decoding
• “Logical” representation from replication stream
– INSERT/UPDATE/DELETE operations
– Can be replayed on different version/platform
• pg_recvlogical command
– Shows how it works
• Replication can be more flexible
– BDR (Bi-Directional Rep.), Slony, and more ...
– Continuous Backup as well

41. pg_recvlogical

42. Administration
(ALTER SYSTEM)

43. Administration - ALTER
SYSTEM
• ALTER SYSTEM SET
– puts new value in postgresql.auto.conf
– pg_reload_conf() reloads them.
– postgresql.auto.conf takes priority over
postgresql.conf.
• ALTER SYSTEM RESET
– Remove values from postgresql.auto.conf.

44. Infrastructure
(For Parallelization)

45. Dynamic Background
Workers
• In 9.3, background workers must start at the
postmaster (listener process) startup.
• After 9.4, they can be launched “on-demand”
basis.
• From parallelization point of view...
– It allows to launch multiple background
processes to execute child queries in parallel.

46. Dynamic Shared Memory
• Shared memory can be allocated “on-demand”
basis
– Cf.) by background workers
• Main segment (ex. shared_buffers) still fixed at
startup
• Also supports lightweight message queue
• From parallelization point of view...
– It allows to share data and communicate with
several bgworker processes.

47. My Tiny Favorite
(pl/pgsql stacktrace)

48. pl/pgsql stacktrace
http://h50146.www5.hp.com/services/ci/opensource/pdfs/PostgreSQL_9_4%20_Ver_1_0.pdf

49. There are many other
enhancements,
so please try it asap.

50. Beyond 9.5

52. Commitfest 2015-7~
CommitFest is a process to review, fix and commit
the submitted patches.
• Parallel Seq scan
• Waits monitoring
• Support multiple synchronous standby servers
• and others..
Still work in progress...
commitfest.postgresql.org

53. Wrap-up
• One of the most developer-friendly
RDBMSes in the world.
• Analytics features and the performance
are improving.
• Things are going to parallel.

54. Resources
• www.postgresql.org
• www.planetpostgresql.org
• www.pgcon.org
• wiki.postgresql.org
• www.postgresql.org/docs/9.5
• news.ycombinator.com/item?id=10039527

55. Postgres Toolkit
• A script collection to manage PostgreSQL
– Helps DBA to perform complicated tasks
– Consists of 13 scripts as of v0.2.2.
• A "Victorinox" for PostgreSQL DBA
• uptime.jp/go/pt

56. SQL Firewall for Postgres
SQL Injection prevented!

57. pgDay Asia 2016
• As A Joint Event with FOSSASIA 2016
– 1 Day, 2 Tracks (not fixed yet)
• FOSSASIA 2016
– March 18h-20th in Singapore
• Still “Work In Progress”, but mark your
calendar NOW!
Photo by Michael Cannon https://flic.kr/p/rieAXe

58. Any Question?
• E-mail: snaga@uptime.jp
• Twitter, Github: @snaga
• WeChat: satoshinagayasu

59. Thank You!