Postgres has the unique ability to act as a powerful data aggregator or information hub in many IT centers bringing together data from different databases and in different formats.
This presentation reviews Postgres' extensibility, foreign data wrappers, and ability to work with structured relational and unstructured NoSQL-like information such as documents and key-value data.
The Postgres capabilities are unrivaled in enabling a complete view of customers or businesses, analyzing disparate data together, and breaking down data silos within the enterprise.
If you would like to listen to the recording please visit EnterpriseDB > Resources > Webcasts > Ondemand Webcasts.
To speak to someone about EnterpriseDB's solutions and services please email sales@enterprisedb.com.
1. Making Postgres Central in Your Data Center
BRUCE MOMJIAN
January, 2015
This talk explores why Postgres is uniquely capable of functioning
as a central database in enterprises. Title concept from Josh Berkus
Creative Commons Attribution License http://momjian.us/presentations
1 / 44
2. PostgreSQL the database…
◮ Open Source Object Relational DBMS since 1996
◮ Distributed under the PostgreSQL License
◮ Similar technical heritage as Oracle, SQL Server & DB2
◮ However, a strong adherence to standards (ANSI-SQL 2008)
◮ Highly extensible and adaptable design
◮ Languages, indexing, data types, etc.
◮ E.g. PostGIS, JSONB, SQL/MED
◮ Extensive use throughout the world for applications and
organizations of all types
◮ Bundled into Red Hat Enterprise Linux, Ubuntu, CentOS
and Amazon Linux
Making Postgres Central in Your Data Center 2 / 44
3. PostgreSQL the community…
◮ Independent community led by a Core Team of six
◮ Large, active and vibrant community
◮ www.postgresql.org
◮ Downloads, Mailing lists, Documentation
◮ Sponsors sampler:
◮ Google, Red Hat, VMWare, Skype, Salesforce, HP and
EnterpriseDB
◮ http://www.postgresql.org/community/
Making Postgres Central in Your Data Center 3 / 44
4. EnterpriseDB the company…
◮ The worldwide leader of Postgres based products and
services founded in 2004
◮ Customers include 50 of the Fortune 500 and 98 of the
Forbes Global 2000
◮ Enterprise offerings:
◮ PostgreSQL Support, Services and Training
◮ Postgres Plus Advanced Server with Oracle Compatibility
◮ Tools for Monitoring, Replication, HA, Backup & Recovery
Community
◮ Citizenship
◮ Contributor of key features: Materialized Views, JSON, &
more
◮ Nine community members on staff
Making Postgres Central in Your Data Center 4 / 44
7. 1. Object-Relational (Extensibility)
Object-relational databases like Postgres support support classes
and inheritance, but most importantly, they define database
functionality as objects that can be easily manipulated.
http://en.wikipedia.org/wiki/Object-relational_database
Making Postgres Central in Your Data Center 7 / 44
9. Example: ISBN Data Type
CREATE EXTENSION isn;
dT
List of data types
Schema | Name | Description
--------+--------+--------------------------------------------------
public | ean13 | International European Article Number (EAN13)
public | isbn | International Standard Book Number (ISBN)
public | isbn13 | International Standard Book Number 13 (ISBN13)
public | ismn | International Standard Music Number (ISMN)
public | ismn13 | International Standard Music Number 13 (ISMN13)
public | issn | International Standard Serial Number (ISSN)
public | issn13 | International Standard Serial Number 13 (ISSN13)
public | upc | Universal Product Code (UPC)
http://www.postgresql.org/docs/current/static/isn.html
Making Postgres Central in Your Data Center 9 / 44
10. ISBN Behaves Just Like Built-In Types
dTS
…
pg_catalog | integer | -2 billion to 2 billion integer, 4-byte storage
…
public | isbn | International Standard Book Number (ISBN)
Making Postgres Central in Your Data Center 10 / 44
11. The System Catalog Entry for INTEGER
SELECT * FROM pg_type WHERE typname = ’int4’;
-[ RECORD 1 ]--+---------
typname | int4
typnamespace | 11
typowner | 10
typlen | 4
typbyval | t
typtype | b
typcategory | N
typispreferred | f
typisdefined | t
typdelim | ,
typrelid | 0
typelem | 0
typarray | 1007
typinput | int4in
typoutput | int4out
typreceive | int4recv
typsend | int4send
typmodin | -
typmodout | -
typanalyze | -
typalign | i
typstorage | p
typnotnull | f
typbasetype | 0 Making Postgres Central in Your Data Center 11 / 44
12. The System Catalog Entry for ISBN
SELECT * FROM pg_type WHERE typname = ’isbn’;
-[ RECORD 1 ]--+---------------
typname | isbn
typnamespace | 2200
typowner | 10
typlen | 8
typbyval | t
typtype | b
typcategory | U
typispreferred | f
typisdefined | t
typdelim | ,
typrelid | 0
typelem | 0
typarray | 16405
typinput | isbn_in
typoutput | public.isn_out
typreceive | -
typsend | -
typmodin | -
typmodout | -
typanalyze | -
typalign | d
typstorage | p
typnotnull | f
typbasetype | 0 Making Postgres Central in Your Data Center 12 / 44
13. Not Just Data Types, Languages
CREATE EXTENSION plpythonu;
dL
List of languages
Name | Owner | Trusted | Description
-----------+----------+---------+------------------------------------------
plpgsql | postgres | t | PL/pgSQL procedural language
plpythonu | postgres | f | PL/PythonU untrusted procedural language
http://www.postgresql.org/docs/current/static/plpython.html
Making Postgres Central in Your Data Center 13 / 44
14. Available Languages
◮ PL/Java
◮ PL/Perl
◮ PL/pgSQL (like PL/SQL)
◮ PL/PHP
◮ PL/Python.
◮ PL/R (like SPSS)
◮ PL/Ruby
◮ PL/Scheme
◮ PL/sh
◮ PL/Tcl
◮ SPI (C)
http://www.postgresql.org/docs/current/static/external-pl.html
Making Postgres Central in Your Data Center 14 / 44
15. Specialized Indexing Methods
◮ BTree
◮ Hash
◮ GiST (generalized search tree)
◮ SP-GiST (space-partitioned GiST)
◮ GIN (generalized inverted index)
http://www.postgresql.org/docs/current/static/indexam.html
Making Postgres Central in Your Data Center 15 / 44
16. Index Types Are Defined in the System Catalogs Too
SELECT amname FROM pg_am;
amname
--------
btree
hash
gist
gin
spgist
http://www.postgresql.org/docs/current/static/catalog-pg-am.html
Making Postgres Central in Your Data Center 16 / 44
17. Operators Have Similar Flexibility
Operators are function calls with left and right arguments of specified types:
doS
Schema | Name | Left arg type | Right arg type | Result type | Description
…
pg_catalog | + | integer | integer | integer | add
dfS
Schema | Name | Result data type | Argument data types | Type
…
pg_catalog | int4pl | integer | integer, integer | normal
Making Postgres Central in Your Data Center 17 / 44
18. Other Extensibility
◮ Casts are defined in pg_cast, int4(float8)
◮ Aggregates are defined in pg_aggregate, sum(int4)
Making Postgres Central in Your Data Center 18 / 44
19. Externally Developed Plug-Ins
◮ PostGIS (Geographical Information System)
◮ PL/v8 (server-side JavaScript)
◮ experimentation, e.g. full text search was originally
externally developed
Making Postgres Central in Your Data Center 19 / 44
20. Offshoots of Postgres
◮ AsterDB
◮ Greenplum
◮ Informix
◮ Netezza
◮ ParAccel
◮ Postgres XC
◮ Redshift (Amazon)
◮ Truviso
◮ Vertica
◮ Yahoo! Everest
https://wiki.postgresql.org/wiki/PostgreSQL_derived_databases
http://de.slideshare.net/pgconf/elephant-roads-a-tour-of-postgres-forks
Making Postgres Central in Your Data Center 20 / 44
22. Plug-In Is Not a Bad Word
Many databases treat extensions as special cases, with serious
limitations. Postgres built-ins use the same API as extensions, so ll
extensions operate just like built-in functionality.
Making Postgres Central in Your Data Center 22 / 44
23. Extensions and Built-In Facilities
Behave the Same
Extensions
PL/R
ISN
PostGIS
Postgres System Tables
int4
btree
sum()
PL/pgSQL
Making Postgres Central in Your Data Center 23 / 44
25. NoSQL Types
There is no single NoSQL technology. They all take different
approaches and have different features and drawbacks:
◮ Key-Value stores, e.g. Redis
◮ Document databases, e.g. MongoDB (JSON)
◮ Columnar stores: Cassandra
◮ Graph databases: Neo4j
Making Postgres Central in Your Data Center 25 / 44
26. Why NoSQL Exists
Generally, NoSQL is optimized for:
◮ Fast querying
◮ Auto-sharding
◮ Flexible schemas
Making Postgres Central in Your Data Center 26 / 44
27. NoSQL Sacrifices
◮ A powerful query language
◮ A sophisticated query optimizer
◮ Data normalization
◮ Joins
◮ Referential integrity
◮ Durability
Making Postgres Central in Your Data Center 27 / 44
28. Are These Drawbacks Worth the Cost?
◮ Difficult Reporting Data must be brought to the client for
analysis, e.g. no aggregates or data analysis functions.
Schema-less data requires complex client-side knowledge for
processing
◮ Complex Application Design Without powerful query
language and query optimizer, the client software is
responsible for efficiently accessing data and for data
consistency
◮ Durability Administrators are responsible for data retention
Making Postgres Central in Your Data Center 28 / 44
29. When Should NoSQL Be Used?
◮ Massive write scaling is required, more than a single server
can provide
◮ Only simple data access pattern is required
◮ Additional resource allocation for development is acceptable
◮ Strong data retention or transactional guarantees are not
required
◮ Unstructured duplicate data that greatly benefits from
column compression
Making Postgres Central in Your Data Center 29 / 44
30. When Should Relational Storage Be Used?
◮ Easy administration
◮ Variable workloads and reporting
◮ Simplified application development
◮ Strong data retention
Making Postgres Central in Your Data Center 30 / 44
31. The Best of Both Worlds: Postgres
Postgres has many NoSQL features without the drawbacks:
◮ Schema-less data types, with sophisticated indexing support
◮ Transactional schema changes with rapid additional and
removal of columns
◮ Durability by default, but controllable per-table or
per-transaction
◮ Postgres XC and PL/Proxy allow auto-sharding for write
scaling
Making Postgres Central in Your Data Center 31 / 44
32. Schema-Less Data: JSON
CREATE TABLE customer (id SERIAL, data JSON);
INSERT INTO customer VALUES (DEFAULT, ’{"name" : "Bill", "age" : 21}’);
SELECT data->’name’ FROM customer WHERE (data->’age’)::text = ’21’;
?column?
----------
"Bill"
Making Postgres Central in Your Data Center 32 / 44
33. Easy Relational Schema Changes
ALTER TABLE customer ADD COLUMN status CHAR(1);
BEGIN WORK;
ALTER TABLE customer ADD COLUMN debt_limit NUMERIC(10,2);
ALTER TABLE customer ADD COLUMN creation_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE customer RENAME TO cust;
COMMIT;
Making Postgres Central in Your Data Center 33 / 44
34. 3. Data Analytics
◮ Aggregates
◮ Optimizer
◮ Server-side languages, e.g. PL/R
◮ Window functions
◮ Bitmap heap scans
◮ Tablespaces
◮ Data partitioning
◮ Materialized views
◮ Common (recursive) table expressions
◮ Min/Max indexes (coming in 2015)
Data warehouse-specific solutions are required for parallel
operations across servers.
http://www.slideshare.net/PGExperts/really-big-elephants-postgresql-dw-15833438
http://wiki.postgresql.org/images/3/38/PGDay2009-EN-Datawarehousing_with_PostgreSQL.pdf
Making Postgres Central in Your Data Center 34 / 44
35. Read-Only Slaves for Analytics
Network
Data WarehouseMaster Server
/pg_xlog/pg_xlog
Making Postgres Central in Your Data Center 35 / 44
36. 4. Foreign Data Wrappers (Database Federation)
Foreign data wrappers (SQL MED) allow queries to read and write
data to foreign data sources. Foreign database support includes:
◮ CouchDB
◮ Informix
◮ MongoDB
◮ MySQL
◮ Neo4j
◮ Oracle
◮ Postgres
◮ Redis
The transfer of joins, aggregates, and sorts to foreign servers is not yet
implemented.
http://www.postgresql.org/docs/current/static/ddl-foreign-data.html
http://wiki.postgresql.org/wiki/Foreign_data_wrappers
Making Postgres Central in Your Data Center 36 / 44
37. Foreign Data Wrappers to Interfaces
◮ JDBC
◮ LDAP
◮ ODBC
Making Postgres Central in Your Data Center 37 / 44
38. Foreign Data Wrappers to
Non-Traditional Data Sources
◮ Files
◮ HTTP
◮ AWS S3
◮ Twitter
Making Postgres Central in Your Data Center 38 / 44
39. Foreign Data Wrapper Example
CREATE SERVER postgres_fdw_test
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host ’localhost’, dbname ’fdw_test’);
CREATE USER MAPPING FOR PUBLIC
SERVER postgres_fdw_test
OPTIONS (password ’’);
CREATE FOREIGN TABLE other_world (greeting TEXT)
SERVER postgres_fdw_test
OPTIONS (table_name ’world’);
det
List of foreign tables
Schema | Table | Server
--------+-------------+-------------------
public | other_world | postgres_fdw_test
(1 row)
Foreign Postgres server name in red; foreign table name in blue
Making Postgres Central in Your Data Center 39 / 44
40. Read and Read/Write Data Sources
Postgres
ora_tab
tw_tab
mon_tab
MongoDB
Twitter
Oracle
Making Postgres Central in Your Data Center 40 / 44
41. 5. Postgres Centrality
Postgres can rightly take a central place in the data center with
its:
◮ Object-Relation flexibility and extensibility
◮ NoSQL-like workloads
◮ Powerful data analytics capabilities
◮ Access to foreign data sources
No other database has all of these key components.
Making Postgres Central in Your Data Center 41 / 44
43. Additional Resources…
◮ Postgres Downloads:
◮ www.enterprisedb.com/downloads
◮ Product and Services information:
◮ info@enterprisedb.com
Making Postgres Central in Your Data Center 43 / 44