Transparent Data Encryption in PostgreSQL and Integration with Key Management Service

Copyright©2019 NTT Corp. All Rights Reserved.
,

• Database and Threats
• Data at rest Encryption
• Transparent data encryption
• Transparent data encryption in PostgreSQL
• Key rotation
• Key Management
• Integration of PostgreSQL with key management system
• Conclusion

• Database servers are often the primary target of the
following attacks
• Privilege abuse
• Database SQL injections attacks
• Storage media theft
• Eavesdropping attacks between client and server
• etc.
44
Eavesdropping
attacks
4
4

• Reason to protect database
• Databases store valuable and sensitive data, some leakage also
causes a high risk
• Essential data protection standards and regulations for
businesses
• PCI DSS, GDPR, HIPPA etc.
Various data protection standards and regulations
require data encryption

• Data at rest
• Backup files and database cluster files stored in physical storage
• Measures against threats to data at rest in PostgreSQL
• Data encryption using pgcrypto or Full disk encryption
However, using pgcrypto or full disk encryption
does not meet some requirements
Data is
secure
! Data
leak
Threat of
theft occurred
Database
storage
Database
storage
User data stored
encrypted
User data stored
non-encrypted

• Minimize performance degradation
• Using pgcrypto degrades application program performance
• Platform-independent
• Minimize application program development cost
• Secure encryption key management
• Periodic key rotation

• Using full disk encryption depends on platforms

• Using pgcrypto requires many application program modifications

• Many modifications of the application program source
code for areas(Tables) where encryption is required
• Regression testing required after source code
modification
• Encryption supports increase time and labor costs
=# INSERT INTO card_info ( user_name, card_number)
VALUES ( ‘MOON INSUNG’, ‘1234-2345-3456-4567’ );
=# INSERT INTO card_info ( user_name, card_number)
VALUES ( ‘MOON INSUNG’,
encrypt (‘1234-2345-3456-4567’, 'KEY_VALUE', ‘aes-cbc’) );
SQL statement before
encryption
SQL statement after
encryption using pgcrypto

• pgcrypto doesn't provide secure encryption key management
facilities out-of-the-box

• If encryption key is leaked, the encrypted data at rest
cannot be protected from threats of malicious access
• Storing encrypted data and its key in the same place
makes the data encryption meaningless

• Key management in a separate secure place for the
encryption keys
Database storage
Encrypted
data
Threat of
theft occurred
Separate location
Secure Place
Data is
secure

• Reduced performance due to re-encryption when rotating keys
Database
storage
Encrypted
data Decryption
current key
Database
storage
Non-
encrypted
data
Database
storage
Encrypted
dataEncryption
New key
Performance degradation due to re-encryption
and data unavailability

• Requirements related to data encryption
• Difficult to satisfy these requirements related to data
encryption using pgcrypto and full disk encryption in
PostgreSQL
In the following slides,
we describe the result of our efforts to address these challenges

1. Per tablespace encryption
2. Transparent encryption at a layer between
PostgreSQL’s shared buffer and OS
3. 2-tier key architecture
4. WAL encryption
5. System catalogs encryption
6. Temporary files encryption
- - -

• Create encryption-enabled tablespaces
• Tables and indexes on the tablespace are transparently
encrypted
• Less modification of DDLs
.
=# CREATE TABLESPACE enc_tblsp
LOCATION ... WITH
(eyncryption_algorithm = ‘aes128’);
=# CREATE TABLE card (...) TABLESPACE enc_tblsp;
=# SET default_tablespace TO enc_tblsp;
=# CREATE TABLE card2 (..);

• Two possible solutions for now
• Using pgcrypto with views and triggers
• Full disk encryption (FDE)
• Our solution is that encryption and decryption are
performed when writing/reading to/from disk

/ / /
postgres
Shared Buffer
Disk
postgres postgres
Page Cache (Kernel)
raw block data

/ / /
postgres
Disk
postgres postgres
Page Cache (Kernel)
raw block data
Shared Buffer
Backend processes
read pages from the
shared buffers and
modify them.

/ / /
postgres
Disk
postgres postgres
Page Cache (Kernel)
raw block data
Shared Buffer
bgwriter periodically
writes the dirty pages
out to the kernel page
cache.

/ / /
postgres
Disk
postgres postgres
raw block data
Shared Buffer
Page Cache (Kernel)
Dirty pages are
flushed to the disk by
the checkpointer or
the kernel.

-
postgres
Shared Buffer
Disk
Pros:
• Protect data even on
shared buffer
Cons:
• Encryption and
decryption are
needed whenever
accessing buffers on
shared buffer
• Different backends
encrypt/decrypt the
same buffer
postgres postgres
Page Cache (Kernel)
raw data
encrypted data

- .
postgres
Shared Buffer
Disk
Pros:
• Less execution of
encryption and
decryption
Cons:
• Platform dependence
• Cannot protect data
from peeking by
logged-in OS user
postgres postgres
Page Cache (Kernel)
raw data
encrypted data

3 ) ) ) -( 2 .- .3 . 32 .-
postgres
Shared Buffer
Disk
Pros:
• Relatively less execution
of encryption and
decryption
• Prevent peeking file on
disk
Cons:
• Possibly more encryption
and decryption are
performed when
database size > shared
buffer
postgres postgres
Page Cache (Kernel)
raw data
encrypted data

Transparence Performance
impacts
Protection against
threats
1. Using pgcrypto OK
(using views
and triggers)
High * Disk thefts
* Memory dump
* Peeking at data on disk
2. Full disk
encryption
OK Low * Disk thefts
3. Buffer level
encryption
OK Middle * Disk thefts
* Peeking at data on disk

• Wrote proof-of-concept code of per tablespaces TDE
• Tablespace encryption
• Transparent encryption
• 2-tier key architecutre
• Key rotation
• Features the PoC code doesn’t support (for now):
• WAL encryption
• System catalog encryption
• Temporary files encryption

• PostgreSQL 11.1
• Vanilla PostgreSQL
• PostgreSQL with TDE PoC code
• PostgreSQL with pgcrypto
• 32GB RAM, 500GB HDD
• 6GB shared buffers
• Data sets
• 5GB (< shared_buffers)
• 15GB (> shared_buffers)
• Observations
• TPS
• Response times
• Duration: 5min

Latency (90%tile):
vanilla: 1.98 ms, TDE: 2.01 ms,
pgcrypto: 2.28 ms
6000
6500
7000
7500
8000
8500
20
40
60
80
100
120
140
160
180
200
220
240
260
280
300
TPS
Duraiton(sec)
TPS comparison (R:100,W:3)
vanilla tde pgcrypto
8000
8500
9000
9500
10000
10500
11000
10
30
50
70
90
110
130
150
170
190
210
230
250
270
TPS
Duration (sec)
TPS comparison (R:100)
vanilla tde pgcrypto
Latency (90%tile):
vanilla: 2.32 ms, TDE: 2.45 ms,
pgcrypto: 2.66 ms
DB size < shared buffers DB size > shared buffers

• Master Key and Data Encryption Key
• The master key is separated from encrypted data
• Stored outside of the database
• The data encryption keys are managed by database
• Faster key rotation
ENCRYPTED
DATA
Master Key Data Encryption Key
Encrypt/Decrypt
Encrypt/
Decrypt
plain

• Master Key and Data Encryption Key
• The master key is separated from encrypted data
• Stored outside of the database
• The data encryption keys are managed by database
• Faster key rotation
ENCRYPTED
DATA
Master Key Data Encryption Key
Encrypt/Decrypt
Encrypt/
Decrypt
encry
pted
plain
New Master Key
Encrypt/Decrypt

• Key rotation always requires re-encrypting data
• The 2-tier key architecture requires only key rotation of
data encryption keys
• 16, 24, 32 bytes key for AES-128, AES-192, AES-256
• One symmetric key per tablespaces

• WAL is also the sensitive data
• WAL of encrypted relations is encrypted when inserting
to the WAL buffer

• Two system catalogs could have user sensitive data
• pg_statistics
• pg_statistics_ext
.
=# SELECT tablename, attname, histogram_bounds FROM pg_stats
WHERE tablename = 'card';
-[ RECORD 1 ]----+-----------------------------------------------
tablename | card
attname | card_number
histogram_bounds | {1102-6674-6045-5459,1606-6441-9374-1335,2507-
2573-1560-9962,3323-3000-4260-1336,4319-9183-6377-7031,6035-9617-
5940-2060,6682-5210-8901-2679,7304-3837-8200-8185,8391-3583-3888-
1725,9091-3895-2466-7845,9970-5910-3522-1423}

• Temporary files are written bypassing the shared buffers
• base/pgsql_tmp/
• pg_replslots/
.
postgres
Shared Buffer
Disk
temp files

• Per tablespace, buffer-level transparent encryption
• 2-tier key architecture
• Encrypt WAL, system catalogs and temporary files
• SRLU buffer and fork relations are not encrypted
• Pros
• Less DDL modification
• Less performance impact
• Fast key rotation
• Cons
• Cannot set per users
• Cannot prevent attack by malicious super user

• Services or systems that are dedicated to robustly
manage keys
• Usually support some kinds of protocols
• KMIP
• PKCS#11
• SafeNet KeySecure, Amazon KMS, Oracle KeyVault etc
( )( ( (

• Robust key management
• User don’t need to worry about key life cycles
ENCRYPTED
DATA
Master Key
Data Encryption Keys
Get the master key
and decrypt/decrypt
Encrypt/Decrypt
Key Management System
PostgreSQL
Register the master key
Remove an old master key

• KMSs support different interfaces and protocols
• KMIP, PKCS#11, etc.
• Our solution
• Pluggable architecture to communicate with various KMSs
• Add generic key management APIs
• get key, register key, remove key etc.

• Encryption key is also important
• Integration with KMS frees user from key management
• Adding generic key management APIs enable us to
communicate with various key management systems

• Per tablespace, buffer-level transparent data at rest
encryption
• Less performance overhead
• Encrypt WAL, system catalogs and temporary files as well
• 2-tier key architecture
• Fast key rotation
• Integration with KMSs
• More flexible and robust key management

1. Basic components of transparent data encryption
• Per tablespace encryption*
• 2-tier key architecture*
• Key rotation*
• System catalog encryption
• Temporary file encryption
• Fetching the master key by arbitrary commands*
2. WAL encryption
3. Integration with KMSs
• Pluggable
• Registering key, removing key ...
PoC codes has *-marked features

!

Transparent Data Encryption in PostgreSQL and Integration with Key Management Service

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Transparent Data Encryption in PostgreSQL and Integration with Key Management Service

Similar to Transparent Data Encryption in PostgreSQL and Integration with Key Management Service (20)

More from Masahiko Sawada

More from Masahiko Sawada (20)

Recently uploaded

Recently uploaded (20)

Transparent Data Encryption in PostgreSQL and Integration with Key Management Service