Best Practices for a Complete Postgres Enterprise Architecture Setup

© 2015 EnterpriseDB Corporation. All rights reserved. 1
Best Practices for a Complete Postgres
Enterprise Architecture Setup
To watch to the presentation visit -
www.EnterpriseDB.com > Resources > On-Demand Webcasts

•  Introduction to EnterpriseDB
•  Enterprise Data Management Architecture
−  OLTP Infrastructure
−  High Availability
−  Disaster Recovery
−  Data Integration
−  Monitoring and Management
−  Security
•  Summary and Resources
•  Q&A
Agenda
We will take two quick
polls during the webinar to
gather group data on best
practices of interest

POSTGRES
innovation
ENTERPRISE
reliability
24/7
support
Services
& training
Enterprise-class
features, tools &
compatibility
Indemnification
Product
road-map
Control
Thousands
of developers
Fast
development
cycles
Low cost
No vendor
lock-in
Advanced
features
Enabling commercial
adoption of Postgres

EDB Customers
EDB currently has over 2,500 total customers including 50 of the Fortune
500 and 98 of the Forbes Global 2000

EDB is a Industry Leader
Magic Quadrant for Operational DBMS, Q4 ‘14
• Gartner Comments
− “EnterpriseDB is responsible
for many features of
PostgreSQL, contributing to
JSON, materialized views and
partitioning.”
− “Clients report that the
functionality of EnterpriseDB's
Postgres Plus Oracle
Compatibility Feature is now
more than sufficient to run both
mission-critical and non-
mission-critical applications.”
− “Customers commend the
compatibility with Oracle, the
stability of the DBMS and the
product support.”

EDB is an Open Source Community
Leader
Amit Kapila
Ashesh Vashi
Bruce Momjian
Dave Page
Devrim Gunduz
Jan Wieck
Kevin Grittner
Korry Douglas
Muhammad Usama
Robert M Haas
Thom Brown

Where Do Users Need Help
# Tickets Phase
Category POC Dev Deployment Maintenance Grand Total
Connectors 0% 2% 1% 0% 3%
Bug 0% 0% 0% 0% 1%
How to 0% 1% 0% 0% 1%
Product Awareness 0% 1% 0% 0% 1%
Database 10% 9% 17% 37% 73%
Bug 1% 2% 1% 2% 6%
Corruption 0% 0% 0% 2% 2%
Enhancements 0% 0% 0% 0% 1%
How to 4% 3% 6%
10%
23%

Product Awareness 4% 3% 8%
15%
31%

Tuning 0% 0% 1% 7%
9%

Replication 1% 4% 4% 2% 12%
Bug 0% 1% 0% 0% 2%
How to 0% 0% 3% 1% 4%
Tuning 0% 0% 0% 1% 1%
Utilities 1% 3% 6% 3% 12%
Bug 0% 1% 1% 0% 2%
Enhancements 0% 0% 0% 0% 0%
How to 1% 1% 3% 1% 6%
Grand Total 12% 17% 28% 42% 100%
Questions focus on
deploying, supporting
and maintaining
Postgres in the
Enterprise

© 2013 EDB All rights reserved 8.1. 8
Enterprise Architecture with Postgres

•  Effective data center implementations use a reference
architecture to:
−  Accelerate the implementation of a solution
−  Lower operational costs
−  Build-in flexibility
−  Enhance performance
−  Lower complexity
−  Reduce risk
−  Avoid the ‘OMG – MTP’ moment!
•  EnterpriseDB Postgres reference architecture
−  modular and standards-based
−  supports a wide range of enterprise deployments
Why?
OMG!
Tomorrow is
MTP!

•  OLTP Infrastructure
−  ACID Compliant
−  High Performance
−  Scalable
−  Reliable
•  High Availability
−  Protect against hardware
failure
−  Protect against software failure
•  Disaster Recovery
−  Protect against site failure
−  Protect against operator error
•  Data Integration
−  Integration with other Line of
Business systems
−  Heterogeneous integration with
Microsoft, Oracle, etc
•  Monitoring and Management
−  Capacity Planning
−  Event management and
alerting
•  Security
−  Authentication and
Authorization
−  Encryption and data protection
Key Components of a Enterprise Data
Management Architecture

?
Tool and Infrastructure Choices
HighAvailability
Monitoring and Management
DisasterRecovery
Data Integration
Open PostgreSQL Monitoring
pgAdmin
phpPgAdmin
repmgr
HandyRep
PgHA
Slony
Bucardo
LondisteBDR
BARManpg_rman
TeamPostgreSQL
OmniPITR
Circonus
pgBadger
pgFouine
powa
pitrery
wal-e
•  Multiple license models (PostgreSQL, GPL, AGPL)
•  Multiple vendors with different SLAs
•  Overlap and gaps
check_postgres
PITRTools
Nagios XI wizard

EDB Integrated Postgres Portfolio
•  OLTP Infrastructure
−  Postgres Plus Advanced
Server PPAS
−  Oracle Compliant
−  Enhanced partitioning, security
and tooling (Profiler, Index
Advisor, Runtime Statistics)
•  High Availability
−  EDB Failover Manager
−  Robust, cost effective,
commodity hardware
•  Disaster Recovery
−  EDB BART (Backup and
Recovery Tool)
−  Native Streaming Replication
•  Data Integration
−  xDB Replication (Single
Master and Multi Master)
−  Heterogeneous integration with
Oracle, MS SQL Server and
Postgres
•  Monitoring and Management
−  Postgres Enterprise Manager
−  Enterprise level management,
monitoring and alerting for
database and infrastructure
•  Security
−  PPAS Enhancements
−  Virtual Private Database (VPD)
−  Code protection
−  SQL Injection Attack Guard

PEM Client
Oracle/SQL
Server/PG
Replication
DB MasterWitness
Offsite
Replica
Virtual IP
DB Client
Backup &
Recovery
PEM ServerHigh Availability
Disaster Recovery
HA Replica
Read/HA
Replica
Robust,
Resilient,
Scalable
Postgres
Enterprise
Architecture
Monitoring
DataIntegration
Simple,
development
focused setup

OLTP Infrastructure

Security
Tools
Performance
Compatibility
ADVANCED SERVER
POSTGRES
PLUS

Postgres Plus
Advanced
Server
Resource
Manager
(CPU & I/O)
Reporting
Transactions
80%
20%
PPAS 9.4 Resource Manager
•  DBA assigns CPU & I/O to job groups
•  Allocates and prioritizes consumption of resources
•  Low priority jobs don’t hurt high priority jobs

SQL Performance Management
−  SQL Profiler captures a SQL workload and
locates the worst running SQL
−  Both ad-hoc and scheduled operations
supported
−  Provides a number of SQL-based
performance metrics along with EXPLAIN
analysis
−  Integrates with Index Advisor, which
analyzes SQL statements and
recommends new indexes to improve
performance
−  EDB SQL Profiler & Index advisor are
components of EDB Postgres Enterprise
Manager (PEM)
•  Poorly optimized SQL code is the NUMBER ONE cause of
database problems—EDB SQL Profiler & Index Advisor
tools can help:

•  Types of storage systems
−  Direct Attached Storage (DAS)
−  Storage Area Network (SAN) w. Fiber Channel
−  Storage Area Network (SAN) w. iSCSI
−  Network File System (NFS)
•  DAS: fast and low latency
•  SAN: fast, expensive (w. Fiber Channel), scalable, can
include redundancy and smart file system operations
•  NFS: not an optimal solution for PostgreSQL data files
Caveat Emptor: File System and Storage
Subsystem

High Availability System Design with
Streaming Replication

•  Shared Disk
−  Red Hat Cluster Server, Veritas Cluster, …
−  Mostly used for local failover
−  High $$$
•  Native streaming replication-based
−  Available with Postgres 9.X
−  Log (WAL)-based, fast and reliable
−  Replicate the database locally or remote
−  Synchronous or asynchronous
−  Hub & spoke or cascading
•  EDB Failover Manager (EFM)
−  Leverages native streaming replication
−  Integrates ‘Witness’ concept to create quorum and avoid false
alarms
High Availability Options for Postgres

EDB
EDB Failover Manager
creates fault tolerant
database clusters to
minimize downtime when
a master database fails by
keeping data online in
high availability
configurations
FAILOVER
MANAGER Master
Streaming
Replica
Clients
Witness
Network /
Internet
The MASTER is down!
I agree! Take control now!

Disaster Recovery

Backup and Standby – Reality Check
hot
47%
warm
23%
None
18%
cold
12%
Standby Approach
Physical
Only
36%
Logical
Only
29%
Physical
and
Logical
14%
FS
Snapshot
7%
None
14%
Backup Approach
43% of customers could not execute
PITR (prior to EDB Architecture Engagement) –
some had NO backup in place
30% of customers would have been
slow or unable to recover from failure

•  Why do you need backup and recovery? Backup and
recovery strategies protect you in case of:
−  Catastrophic device failure
−  Site failure
−  Maintenance
−  Operator error
−  Compliance
−  Data corruption
•  Multiple components:
−  Logical backup provides granularity in objects (tables, table
spaces, databases)
−  Physical backups provide granularity in time for PITR
−  Cold/off-line backups
−  Hot/on-line backup, with WAL archiving
Backup and Recovery Strategies

•  Backup Strategy Framework
−  Logical backup after structural changes or
major updates of reference/meta data
−  Physical backup (daily…)
−  WAL file archiving
−  Combine backup/recovery and streaming replication
−  Uses replica to offload hot backup and logical backup
•  Consider
−  Allowable PITR timeframe (MTR – Mean Time to Recover) and
allowable data loss (RPO – Recovery Point Objective)
−  Data retention policy
−  Test, test, test
−  Periodic backup validation
Backup and Recovery Best Practices

Disaster Recovery for Postgres
•  Two components
−  Offsite replication
−  Streaming replication to Disaster Recovery site
−  Backup
−  Logical backup
−  Physical backup
−  WAL archiving for PITR

•  Simplifies and reduces errors with a system-wide
catalog and command line tool
•  Online backup and recovery
across local and remote
servers
•  Local and remote online
physical backups
•  Auto-compression and
MD5 checksum verification
•  Continuous WAL Archiving
and PITR
•  Support for tablespace
restoration on different paths
BART Architecture

Database Infrastructure
Management and Monitoring

•  Monitor the health of your server and databases for
proactive management and issue prevention
Monitoring and Management
• What should I monitor?
• How do I reduce downtime?
• How do I know if I need more space?
• How to find bloat in tables & indexes?

Maintenance Highlights from 20+ AHCs
Unused
Indexes
39%
Bloated
Indexes
30%
Bloated Tables
13%
Missing
Indexes
9%
Missing
PK
9%
Bloat management
(43%) is a key
production
problem
specific to
Postgres

•  Utilize alerting and event management across statistics
−  Storage, user activity, connections
−  I/O, memory analysis
−  Database size, tablespace size
−  Session activity - workload, locks, waits
•  Perform regular capacity planning
−  Analyze historical usage statistics of objects
−  Project the anticipated usage statistics for an object
−  Collect and analyze metrics for specific:
−  Host/operating system
−  Postgres server
−  Database
−  Database object (table, index, function etc.)
Effective Monitoring and Capacity Planning

General Monitoring Recommendations
Description Frequency Alerting Criteria / Parameters monitored
Postgres instance is
running and connection
count
Every minute (i) When postgres instance is not running, servers down
(ii) When number of connections reaches percentage of
max_connections
(iii) When number of connections crosses threshold
Load average, disk
space
Every 15
minutes
(i) When load average is above threshold
(ii) When disk consumption percentage above threshold
(iii) When disk space available is below threshold
Long running and
waiting queries in
database
Every 10
minutes
(i) When any query is running for longer than threshold
(ii) When any query is in state of "idle in transaction" for
more than threshold
(iii) When any query is waiting for more than threshold
Bloats for tables and
indexes
Once a day (i)  Total table bloat, highest table bloat
(ii) When database tables and/or indexes individually
bloated more than critical threshold
Vacuum maintenance Once a day (i)  Last vacuum
(ii) Last auto-vacuum
Streaming replication
sync with primary
Every 5
minutes
(i) When lag detected on streaming cluster exceeds
threshold
Standby server lag behind the master by WAL segments
Standby server lag behind the master by WAL pages

•  Consumption projection:
−  At current consumption rates when will I drop below 100MB of disk?
−  At the consumption rate of the past 3 days when will I drop below 1 GB of disk
space?
•  Point in Time activity
−  Users were complaining of performance problems from 3 to 4 PM today. What
was going on in the system, the database, how many users were connected?
−  We get a load spike every day at 11. What was happing at the time of the load
spike?
•  Root cause for slower query performance
−  Queries that access the employ table seem to be taking longer and longer.
View the growth of the number of dead tuples in the table, see what kind of
scans have been executing against the table (index or sequential)
•  Activity monitoring
−  Which database on the server is the most active? What kind of growth is it
seeing?
Capacity Planning and Management –
Use Cases

•  Single management console allows easy visual control
•  Works with both PostgreSQL and Postgres Plus
•  Start/stop, configure, define and manage storage, security and database objects via
single graphical console
MONITOR MANAGE TUNE
EDB Postgres Enterprise Manager (PEM)
Mission critical OS and
database statistics
collection
Monitor real-time alerts
Predefined (200+) and
custom alerts via SMTP
or SNMP
Predefined & custom at-a-
glance global dashboards
Replication monitoring
CRUD operations on all
database objects
Bulk operations across
multiple servers
Historical reports
Capacity Manager for
planning & forecasting
Customizable GUI charts,
tables & graphs
Identify poorly
running SQL
statements
SQL/Profiler to speed
up large workloads
Index Advisor to
suggest and create
indexes
Postgres Expert for
best practice
enforcement
Tuning Wizard for
machine utilization
and load profiles

Unmanaged
Hosts
An efficient distributed
architecture perfectly
suited for managing,
monitoring and tuning
large numbers of Postgres
servers in multiple
locations
Monitoring Data
Enterprise Management
Connections
PEM Clients
or Web
Clients
Managed
Hosts
with PEM
Agents
Enterprise
Manager Agent
Installed on each
managed
machine; collects
data on OS and
database health
and operations PEM
Server
Enterprise
Manager Server
Centralized
storage for agent-
collected data and
client dashboards
Host Database
Management
Connection
Enterprise Manager Clients
User GUI console with global at-a-
glance monitoring dashboards;
used to carry out centralized
database administration and tuning
Platform-specific and web clients
POSTGRES
ENTERPRISE
MANAGER (PEM)
Architecture
EDB

Monitor All Your Postgres Databases From
One Screen
•  Customized global
dashboard
•  View up/down status
of all agents
•  Monitor alerts from
many servers in one
place
•  Navigate to
Dashboards for
further analysis

Data Integration

•  Postgres solutions exist in context with other Line of
Business solutions
•  Options
−  Physical data replication – xDB Replication Engine
−  Virtual data replication – Foreign Data Wrappers
−  ETL (Extract, Transfer, Load)
Data Integration with Postgres

Oracle
SQL Server
PostgreSQL
Advanced Server
READ/WRITE
PostgreSQL
Advanced Server
Data filtering
Scheduling
READ
MASTER
REPLICA
Flexible
§  Heterogeneous
§  Filter only data you need replicated
§  Continuous or scheduled
§  Cascading
Multiple Applications
§  Performance—shift read operations
from master to hot replica Backup
§  Testing/new development copies
§  Migrate data to new systems
EDBxDB REPLICATION
SINGLE MASTER
Near real-time
replication

xDB Graphical UI: Making Replication Easy
Browsing publications
and subscriptions
Setting up new
publications
Configuring new
publications
Adding criteria
Adding subscriptions
Reviewing results

Security

Security
•  Conceptual Layers
−  Authentication
−  Authorization
−  Auditing
−  Data Security
−  SQL Injection Attacks
•  Approach
−  Secure physical access
−  Network access limitation
−  Host access limitation
−  Database access
limitation
−  Data access limitation

“By default, PostgreSQL is probably the most security-aware
database available ...”
--Database Hacker's Handbook
Postgres Plus Advanced Server Security
•  Object level privileges assigned to roles and users
•  Row Level Security (Virtual Private Database)
•  EAL2 Certification (augmented with ALC_FLR.2), NIPRNet, SIPRNet, JWICS
•  Stored procedure obfuscation
•  Protection against SQL injection attacks
•  Kerberos and LDAP authentication
•  SQL USAGE privilege and VIEW Security Barriers
•  SSL communication
•  Data Level Encryption (AES, 3DES, etc.)
•  Ability to utilize 3rd party Key Stores in a full PKI Infrastructure
•  Foundation for full compliance with the strictest of security standards (PCI Data
Security Standard)

Security Best Practices for PostgreSQLSecurity Best Practices for PostgreSQL and Postgres Plus Advanced Server
Executive Summary ................................................................3
Introduction ….........................................................................4
Postgres Security Features within the Above Framework ......6
Authentication .........................................................6
Authorization ….......................................................7
Accounting/Auditing ................................................8
Data Security ….......................................................9
SQL Injection Attacks ............................................10
Postgres Plus Advanced Server Security Features ..............11
Further Reading and Useful Links ........................................12
About EnterpriseDB ..............................................................12
Table of Contents
http://info.enterprisedb.com/rs/enterprisedb/images/
Whitepapers_Security_BP_PostgreSQL_and_Postgres_Plus_AS.pdf

•  A successful Postgres implementation is
about more than using Postgres
•  It includes
−  High Availability
−  Disaster Recovery
−  Backup Strategy
−  Integration Strategy
−  Monitoring and Management
−  Security
Summary

PEM Client
Oracle/SQL
Server/PG
Replication
DB MasterWitness
Offsite
Replica
Virtual IP
DB Client
Backup &
Recovery
Disaster Recovery
HA Replica
Read/HA
Replica
Monitoring
DataIntegration
EDB
Failover
Manager
EDB
Backup
and
Recovery
xDB
Replication
Postgres
Enterprise
Manager
Postgres
Streaming
Replication

Best Practices for a Complete Postgres Enterprise Architecture Setup

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Best Practices for a Complete Postgres Enterprise Architecture Setup

Similar to Best Practices for a Complete Postgres Enterprise Architecture Setup (20)

More from EDB

More from EDB (20)

Recently uploaded

Recently uploaded (20)

Best Practices for a Complete Postgres Enterprise Architecture Setup