SlideShare a Scribd company logo
1 of 41
Download to read offline
PostgreSQL worst practices
at FOSDEM PGDay Brussels 2017
Ilya Kosmodemiansky
ik@postgresql-consulting.com
Best practices are just boring
• Never follow them, try worst practices
• Only those practices can really help you to screw the things up
most effectively
• PostgreSQL consultants are nice people, so try to make them
happy
How it works?
• I have a list, a little bit more than 100 worst practices
• I do not make this stuff up, all of them are real-life examples
• I reshuffle my list every time before presenting and extract
some amount of examples
• Well, there are some things, which I like more or less, so it is
not a very honest shuffle
0. Do not use indexes (a test one!)
• Basically, there is no difference between full table scan and
index scan
• You can check that. Just insert 10 rows into a test table on
your test server and compare.
• Nobody deals with more than 10 row tables in production!
1. Use ORM
• All databases share the same syntax
• You must write database-independent code
• Are there any benefits, which are based on database specific
features?
• It always good to learn a new complicated technology
2. Move joins to your application
• Just select * a couple of tables into the application written in
your favorite programming language
• Than join them at the application level
2. Move joins to your application
• Just select * a couple of tables into the application written in
your favorite programming language
• Than join them at the application level
• Now you only need to implement nested loop join, hash join
and merge join as well as query optimizer and page cache
3. Be in trend, be schema-less
• You do not need to design the schema
• You need only one table, two columns: id bigserial and extra
jsonb
• JSONB datatype is pretty effective in PostgreSQL, you can
search in it just like in a well-structured table
• Even if you put a 100M of JSON in it
• Even if you have 1000+ tps
4. Be agile, use EAV
• You need only 3 tables: entity, attribute, value
4. Be agile, use EAV
• You need only 3 tables: entity, attribute, value
• At some point add the 4th: attribute_type
4. Be agile, use EAV
• You need only 3 tables: entity, attribute, value
• At some point add the 4th: attribute_type
• Whet it starts to work slow, just call those four tables The
Core and add 1000+ tables with denormalized data
4. Be agile, use EAV
• You need only 3 tables: entity, attribute, value
• At some point add the 4th: attribute_type
• Whet it starts to work slow, just call those four tables The
Core and add 1000+ tables with denormalized data
• If it is not enough, you can always add value_version
5. Try to create as many indexes as you can
• Indexes consume no disk space
• Indexes consume no shared_bufers
• There is no overhead on DML if one and every column in a
table covered with bunch of indexes
• Optimizer will definitely choose your index once you created it
• Keep calm and create more indexes
6. Always keep all your time series data
• Time series data like tables with logs or session history should
be never deleted, aggregated or archived, you always need to
keep it all
6. Always keep all your time series data
• Time series data like tables with logs or session history should
be never deleted, aggregated or archived, you always need to
keep it all
• You will always know where to check, if you run out of disk
space
6. Always keep all your time series data
• Time series data like tables with logs or session history should
be never deleted, aggregated or archived, you always need to
keep it all
• You will always know where to check, if you run out of disk
space
• You can always call that Big Data
6. Always keep all your time series data
• Time series data like tables with logs or session history should
be never deleted, aggregated or archived, you always need to
keep it all
• You will always know where to check, if you run out of disk
space
• You can always call that Big Data
• Solve the problem using partitioning... one partition for an
hour or for a minute
7. Turn autovacuum off
• It is quite auxiliary process, you can easily stop it
• There is no problem at all to have 100Gb data in a database
which is 1Tb in size
• 2-3Tb RAM servers are cheap, IO is a fastest thing in modern
computing
• Besides of that, everyone likes BigData
8. Keep master and slave on different hardware
• That will maximize the possibility of unsuccessful failover
8. Keep master and slave on different hardware
• That will maximize the possibility of unsuccessful failover
• To make things worser, you can change only slave-related
parameters at slave, leaving defaults for shared_buffers etc.
9. Put a synchronous replica to remote DC
• Indeed! That will maximize availability!
9. Put a synchronous replica to remote DC
• Indeed! That will maximize availability!
• Especially, if you put the replica to another continent
10. Reinvent Slony
• If you need some data replication to another database, try to
implement it from scratch
10. Reinvent Slony
• If you need some data replication to another database, try to
implement it from scratch
• That allows you to run into all problems, PostgreSQL have
had since introducing Slony
11. Use as many count(*) as you can
• Figure 301083021830123921 is very informative for the end
user
• If it changes in a second to 30108302894839434020, it is still
informative
• select count(*) from sometable is a quite light-weighted query
• Tuple estimation from pg_catalog can never be precise
enough for you
12. Never use graphical monitoring
• You do not need graphs
• Because it is an easy task to guess what was happened
yesterday at 2 a.m. using command line and grep only
13. Never use Foreign Keys
(Use local produced instead!)
• Consistency control at application level always works as
expected
• You will never get data inconsistency without constraints
• Even if you already have a bullet proof framework to maintain
consistency, could it be good enough reason to use it?
14. Always use text type for all columns
• It is always fun to reimplement date or ip validation in your
code
• You will never mistakenly convert ”12-31-2015 03:01AM” to
”15:01 12 of undef 2015” using text fields
15. Always use improved ”PostgreSQL”
• Postgres is not a perfect database and you are smart
• All that annoying MVCC staff, 32 bit xid and autovacuum
nightmare look like they look because hackers are oldschool
and lazy
• Hack it in a hard way, do not bother yourself with submitting
your patch to the community, just put it into production
• It is easy to maintain such production and keep it compatible
with ”not perfect” PostgreSQL upcoming versions
16. Postgres likes long transactions
• Always call external services from stored procedures (like
sending emails)
16. Postgres likes long transactions
• Always call external services from stored procedures (like
sending emails)
• Oh, it is arguable... It can be, if 100% of developers were
familiar with word timeout
16. Postgres likes long transactions
• Always call external services from stored procedures (like
sending emails)
• Oh, it is arguable... It can be, if 100% of developers were
familiar with word timeout
• Anyway, you can just start transaction and go away for
weekend
17. Load your data to PostgreSQL in a smart manner
• Write your own loader, 100 parallel threads minimum
17. Load your data to PostgreSQL in a smart manner
• Write your own loader, 100 parallel threads minimum
• Never use COPY - it is specially designed for the task
18. Even if you want to backup your database...
• Use replication instead of backup
18. Even if you want to backup your database...
• Use replication instead of backup
• Use pg_dump instead of backup
18. Even if you want to backup your database...
• Use replication instead of backup
• Use pg_dump instead of backup
• Write your own backup script
18. Even if you want to backup your database...
• Use replication instead of backup
• Use pg_dump instead of backup
• Write your own backup script
• As complicated as possible, combine all external tools you
know
18. Even if you want to backup your database...
• Use replication instead of backup
• Use pg_dump instead of backup
• Write your own backup script
• As complicated as possible, combine all external tools you
know
• Never perform a test recovery
Do not forget
That was WORST practice talk
Questions or ideas? Share your story!
ik@postgresql-consulting.com
(I’am preparing this talk to be open sourced)

More Related Content

What's hot

PostgreSQL: Joining 1 million tables
PostgreSQL: Joining 1 million tablesPostgreSQL: Joining 1 million tables
PostgreSQL: Joining 1 million tablesHans-Jürgen Schönig
 
Why you should care about data layout in the file system with Cheng Lian and ...
Why you should care about data layout in the file system with Cheng Lian and ...Why you should care about data layout in the file system with Cheng Lian and ...
Why you should care about data layout in the file system with Cheng Lian and ...Databricks
 
PostgreSQL_ Up and Running_ A Practical Guide to the Advanced Open Source Dat...
PostgreSQL_ Up and Running_ A Practical Guide to the Advanced Open Source Dat...PostgreSQL_ Up and Running_ A Practical Guide to the Advanced Open Source Dat...
PostgreSQL_ Up and Running_ A Practical Guide to the Advanced Open Source Dat...MinhLeNguyenAnh2
 
Hive on Spark を活用した高速データ分析 - Hadoop / Spark Conference Japan 2016
Hive on Spark を活用した高速データ分析 - Hadoop / Spark Conference Japan 2016Hive on Spark を活用した高速データ分析 - Hadoop / Spark Conference Japan 2016
Hive on Spark を活用した高速データ分析 - Hadoop / Spark Conference Japan 2016Nagato Kasaki
 
MariaDB Performance Tuning and Optimization
MariaDB Performance Tuning and OptimizationMariaDB Performance Tuning and Optimization
MariaDB Performance Tuning and OptimizationMariaDB plc
 
PostgreSQL worst practices, version PGConf.US 2017 by Ilya Kosmodemiansky
PostgreSQL worst practices, version PGConf.US 2017 by Ilya KosmodemianskyPostgreSQL worst practices, version PGConf.US 2017 by Ilya Kosmodemiansky
PostgreSQL worst practices, version PGConf.US 2017 by Ilya KosmodemianskyPostgreSQL-Consulting
 
Developing Scylla Applications: Practical Tips
Developing Scylla Applications: Practical TipsDeveloping Scylla Applications: Practical Tips
Developing Scylla Applications: Practical TipsScyllaDB
 
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Brian Brazil
 
Presto User & Admin Guide
Presto User & Admin GuidePresto User & Admin Guide
Presto User & Admin GuideJEONGPHIL HAN
 
Histograms in MariaDB, MySQL and PostgreSQL
Histograms in MariaDB, MySQL and PostgreSQLHistograms in MariaDB, MySQL and PostgreSQL
Histograms in MariaDB, MySQL and PostgreSQLSergey Petrunya
 
How to be Successful with Scylla
How to be Successful with ScyllaHow to be Successful with Scylla
How to be Successful with ScyllaScyllaDB
 
Improving Presto performance with Alluxio at TikTok
Improving Presto performance with Alluxio at TikTokImproving Presto performance with Alluxio at TikTok
Improving Presto performance with Alluxio at TikTokAlluxio, Inc.
 
PostgreSQL Administration for System Administrators
PostgreSQL Administration for System AdministratorsPostgreSQL Administration for System Administrators
PostgreSQL Administration for System AdministratorsCommand Prompt., Inc
 
Performance Monitoring: Understanding Your Scylla Cluster
Performance Monitoring: Understanding Your Scylla ClusterPerformance Monitoring: Understanding Your Scylla Cluster
Performance Monitoring: Understanding Your Scylla ClusterScyllaDB
 
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PGConf APAC
 
PostgreSQL Performance Tuning
PostgreSQL Performance TuningPostgreSQL Performance Tuning
PostgreSQL Performance Tuningelliando dias
 
Scylla Summit 2022: Scylla 5.0 New Features, Part 1
Scylla Summit 2022: Scylla 5.0 New Features, Part 1Scylla Summit 2022: Scylla 5.0 New Features, Part 1
Scylla Summit 2022: Scylla 5.0 New Features, Part 1ScyllaDB
 

What's hot (20)

PostgreSQL: Joining 1 million tables
PostgreSQL: Joining 1 million tablesPostgreSQL: Joining 1 million tables
PostgreSQL: Joining 1 million tables
 
Why you should care about data layout in the file system with Cheng Lian and ...
Why you should care about data layout in the file system with Cheng Lian and ...Why you should care about data layout in the file system with Cheng Lian and ...
Why you should care about data layout in the file system with Cheng Lian and ...
 
PostgreSQL_ Up and Running_ A Practical Guide to the Advanced Open Source Dat...
PostgreSQL_ Up and Running_ A Practical Guide to the Advanced Open Source Dat...PostgreSQL_ Up and Running_ A Practical Guide to the Advanced Open Source Dat...
PostgreSQL_ Up and Running_ A Practical Guide to the Advanced Open Source Dat...
 
Hive on Spark を活用した高速データ分析 - Hadoop / Spark Conference Japan 2016
Hive on Spark を活用した高速データ分析 - Hadoop / Spark Conference Japan 2016Hive on Spark を活用した高速データ分析 - Hadoop / Spark Conference Japan 2016
Hive on Spark を活用した高速データ分析 - Hadoop / Spark Conference Japan 2016
 
PostgreSQL replication
PostgreSQL replicationPostgreSQL replication
PostgreSQL replication
 
MariaDB Performance Tuning and Optimization
MariaDB Performance Tuning and OptimizationMariaDB Performance Tuning and Optimization
MariaDB Performance Tuning and Optimization
 
PostgreSQL worst practices, version PGConf.US 2017 by Ilya Kosmodemiansky
PostgreSQL worst practices, version PGConf.US 2017 by Ilya KosmodemianskyPostgreSQL worst practices, version PGConf.US 2017 by Ilya Kosmodemiansky
PostgreSQL worst practices, version PGConf.US 2017 by Ilya Kosmodemiansky
 
Developing Scylla Applications: Practical Tips
Developing Scylla Applications: Practical TipsDeveloping Scylla Applications: Practical Tips
Developing Scylla Applications: Practical Tips
 
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
 
Presto User & Admin Guide
Presto User & Admin GuidePresto User & Admin Guide
Presto User & Admin Guide
 
5 Steps to PostgreSQL Performance
5 Steps to PostgreSQL Performance5 Steps to PostgreSQL Performance
5 Steps to PostgreSQL Performance
 
Histograms in MariaDB, MySQL and PostgreSQL
Histograms in MariaDB, MySQL and PostgreSQLHistograms in MariaDB, MySQL and PostgreSQL
Histograms in MariaDB, MySQL and PostgreSQL
 
How to be Successful with Scylla
How to be Successful with ScyllaHow to be Successful with Scylla
How to be Successful with Scylla
 
Postgresql tutorial
Postgresql tutorialPostgresql tutorial
Postgresql tutorial
 
Improving Presto performance with Alluxio at TikTok
Improving Presto performance with Alluxio at TikTokImproving Presto performance with Alluxio at TikTok
Improving Presto performance with Alluxio at TikTok
 
PostgreSQL Administration for System Administrators
PostgreSQL Administration for System AdministratorsPostgreSQL Administration for System Administrators
PostgreSQL Administration for System Administrators
 
Performance Monitoring: Understanding Your Scylla Cluster
Performance Monitoring: Understanding Your Scylla ClusterPerformance Monitoring: Understanding Your Scylla Cluster
Performance Monitoring: Understanding Your Scylla Cluster
 
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs
 
PostgreSQL Performance Tuning
PostgreSQL Performance TuningPostgreSQL Performance Tuning
PostgreSQL Performance Tuning
 
Scylla Summit 2022: Scylla 5.0 New Features, Part 1
Scylla Summit 2022: Scylla 5.0 New Features, Part 1Scylla Summit 2022: Scylla 5.0 New Features, Part 1
Scylla Summit 2022: Scylla 5.0 New Features, Part 1
 

Similar to PostgreSQL worst practices, version FOSDEM PGDay 2017 by Ilya Kosmodemiansky

Unbreaking Your Django Application
Unbreaking Your Django ApplicationUnbreaking Your Django Application
Unbreaking Your Django ApplicationOSCON Byrum
 
Lightening Talk - PostgreSQL Worst Practices
Lightening Talk - PostgreSQL Worst PracticesLightening Talk - PostgreSQL Worst Practices
Lightening Talk - PostgreSQL Worst PracticesPGConf APAC
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tigerElizabeth Smith
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tigerElizabeth Smith
 
Austin Python Learners Meetup - Everything you need to know about programming...
Austin Python Learners Meetup - Everything you need to know about programming...Austin Python Learners Meetup - Everything you need to know about programming...
Austin Python Learners Meetup - Everything you need to know about programming...Danny Mulligan
 
Lessons PostgreSQL learned from commercial databases, and didn’t
Lessons PostgreSQL learned from commercial databases, and didn’tLessons PostgreSQL learned from commercial databases, and didn’t
Lessons PostgreSQL learned from commercial databases, and didn’tPGConf APAC
 
Best practices with development of enterprise-scale SharePoint solutions - Pa...
Best practices with development of enterprise-scale SharePoint solutions - Pa...Best practices with development of enterprise-scale SharePoint solutions - Pa...
Best practices with development of enterprise-scale SharePoint solutions - Pa...SPC Adriatics
 
Keeping MongoDB Data Safe
Keeping MongoDB Data SafeKeeping MongoDB Data Safe
Keeping MongoDB Data SafeTony Tam
 
Neo4j Training Cypher
Neo4j Training CypherNeo4j Training Cypher
Neo4j Training CypherMax De Marzi
 
Geek Sync | Top 5 Tips to Keep Always On Always Humming and Users Happy
Geek Sync | Top 5 Tips to Keep Always On Always Humming and Users HappyGeek Sync | Top 5 Tips to Keep Always On Always Humming and Users Happy
Geek Sync | Top 5 Tips to Keep Always On Always Humming and Users HappyIDERA Software
 
Entity framework advanced
Entity framework advancedEntity framework advanced
Entity framework advancedUsama Nada
 
Building Big Data Streaming Architectures
Building Big Data Streaming ArchitecturesBuilding Big Data Streaming Architectures
Building Big Data Streaming ArchitecturesDavid Martínez Rego
 
The 5 Minute MySQL DBA
The 5 Minute MySQL DBAThe 5 Minute MySQL DBA
The 5 Minute MySQL DBAIrawan Soetomo
 
Generating Sequences with Deep LSTMs & RNNS in julia
Generating Sequences with Deep LSTMs & RNNS in juliaGenerating Sequences with Deep LSTMs & RNNS in julia
Generating Sequences with Deep LSTMs & RNNS in juliaAndre Pemmelaar
 
Internet of Things, TYBSC IT, Semester 5, Unit IV
Internet of Things, TYBSC IT, Semester 5, Unit IVInternet of Things, TYBSC IT, Semester 5, Unit IV
Internet of Things, TYBSC IT, Semester 5, Unit IVArti Parab Academics
 
Reading Notes : the practice of programming
Reading Notes : the practice of programmingReading Notes : the practice of programming
Reading Notes : the practice of programmingJuggernaut Liu
 
Know thy cost (or where performance problems lurk)
Know thy cost (or where performance problems lurk)Know thy cost (or where performance problems lurk)
Know thy cost (or where performance problems lurk)Oren Eini
 
7 Database Mistakes YOU Are Making -- Linuxfest Northwest 2019
7 Database Mistakes YOU Are Making -- Linuxfest Northwest 20197 Database Mistakes YOU Are Making -- Linuxfest Northwest 2019
7 Database Mistakes YOU Are Making -- Linuxfest Northwest 2019Dave Stokes
 

Similar to PostgreSQL worst practices, version FOSDEM PGDay 2017 by Ilya Kosmodemiansky (20)

Unbreaking Your Django Application
Unbreaking Your Django ApplicationUnbreaking Your Django Application
Unbreaking Your Django Application
 
Lightening Talk - PostgreSQL Worst Practices
Lightening Talk - PostgreSQL Worst PracticesLightening Talk - PostgreSQL Worst Practices
Lightening Talk - PostgreSQL Worst Practices
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tiger
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tiger
 
Software + Babies
Software + BabiesSoftware + Babies
Software + Babies
 
Austin Python Learners Meetup - Everything you need to know about programming...
Austin Python Learners Meetup - Everything you need to know about programming...Austin Python Learners Meetup - Everything you need to know about programming...
Austin Python Learners Meetup - Everything you need to know about programming...
 
Lessons PostgreSQL learned from commercial databases, and didn’t
Lessons PostgreSQL learned from commercial databases, and didn’tLessons PostgreSQL learned from commercial databases, and didn’t
Lessons PostgreSQL learned from commercial databases, and didn’t
 
Best practices with development of enterprise-scale SharePoint solutions - Pa...
Best practices with development of enterprise-scale SharePoint solutions - Pa...Best practices with development of enterprise-scale SharePoint solutions - Pa...
Best practices with development of enterprise-scale SharePoint solutions - Pa...
 
Keeping MongoDB Data Safe
Keeping MongoDB Data SafeKeeping MongoDB Data Safe
Keeping MongoDB Data Safe
 
Neo4j Training Cypher
Neo4j Training CypherNeo4j Training Cypher
Neo4j Training Cypher
 
Geek Sync | Top 5 Tips to Keep Always On Always Humming and Users Happy
Geek Sync | Top 5 Tips to Keep Always On Always Humming and Users HappyGeek Sync | Top 5 Tips to Keep Always On Always Humming and Users Happy
Geek Sync | Top 5 Tips to Keep Always On Always Humming and Users Happy
 
Entity framework advanced
Entity framework advancedEntity framework advanced
Entity framework advanced
 
Gpgpu intro
Gpgpu introGpgpu intro
Gpgpu intro
 
Building Big Data Streaming Architectures
Building Big Data Streaming ArchitecturesBuilding Big Data Streaming Architectures
Building Big Data Streaming Architectures
 
The 5 Minute MySQL DBA
The 5 Minute MySQL DBAThe 5 Minute MySQL DBA
The 5 Minute MySQL DBA
 
Generating Sequences with Deep LSTMs & RNNS in julia
Generating Sequences with Deep LSTMs & RNNS in juliaGenerating Sequences with Deep LSTMs & RNNS in julia
Generating Sequences with Deep LSTMs & RNNS in julia
 
Internet of Things, TYBSC IT, Semester 5, Unit IV
Internet of Things, TYBSC IT, Semester 5, Unit IVInternet of Things, TYBSC IT, Semester 5, Unit IV
Internet of Things, TYBSC IT, Semester 5, Unit IV
 
Reading Notes : the practice of programming
Reading Notes : the practice of programmingReading Notes : the practice of programming
Reading Notes : the practice of programming
 
Know thy cost (or where performance problems lurk)
Know thy cost (or where performance problems lurk)Know thy cost (or where performance problems lurk)
Know thy cost (or where performance problems lurk)
 
7 Database Mistakes YOU Are Making -- Linuxfest Northwest 2019
7 Database Mistakes YOU Are Making -- Linuxfest Northwest 20197 Database Mistakes YOU Are Making -- Linuxfest Northwest 2019
7 Database Mistakes YOU Are Making -- Linuxfest Northwest 2019
 

More from PostgreSQL-Consulting

Linux IO internals for database administrators (SCaLE 2017 and PGDay Nordic 2...
Linux IO internals for database administrators (SCaLE 2017 and PGDay Nordic 2...Linux IO internals for database administrators (SCaLE 2017 and PGDay Nordic 2...
Linux IO internals for database administrators (SCaLE 2017 and PGDay Nordic 2...PostgreSQL-Consulting
 
Linux internals for Database administrators at Linux Piter 2016
Linux internals for Database administrators at Linux Piter 2016Linux internals for Database administrators at Linux Piter 2016
Linux internals for Database administrators at Linux Piter 2016PostgreSQL-Consulting
 
Autovacuum, explained for engineers, new improved version PGConf.eu 2015 Vienna
Autovacuum, explained for engineers, new improved version PGConf.eu 2015 ViennaAutovacuum, explained for engineers, new improved version PGConf.eu 2015 Vienna
Autovacuum, explained for engineers, new improved version PGConf.eu 2015 ViennaPostgreSQL-Consulting
 
PostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQPostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQPostgreSQL-Consulting
 
Как PostgreSQL работает с диском
Как PostgreSQL работает с дискомКак PostgreSQL работает с диском
Как PostgreSQL работает с дискомPostgreSQL-Consulting
 
Максим Богук. Postgres-XC
Максим Богук. Postgres-XCМаксим Богук. Postgres-XC
Максим Богук. Postgres-XCPostgreSQL-Consulting
 
Илья Космодемьянский. Использование очередей асинхронных сообщений с PostgreSQL
Илья Космодемьянский. Использование очередей асинхронных сообщений с PostgreSQLИлья Космодемьянский. Использование очередей асинхронных сообщений с PostgreSQL
Илья Космодемьянский. Использование очередей асинхронных сообщений с PostgreSQLPostgreSQL-Consulting
 

More from PostgreSQL-Consulting (10)

Linux IO internals for database administrators (SCaLE 2017 and PGDay Nordic 2...
Linux IO internals for database administrators (SCaLE 2017 and PGDay Nordic 2...Linux IO internals for database administrators (SCaLE 2017 and PGDay Nordic 2...
Linux IO internals for database administrators (SCaLE 2017 and PGDay Nordic 2...
 
Linux internals for Database administrators at Linux Piter 2016
Linux internals for Database administrators at Linux Piter 2016Linux internals for Database administrators at Linux Piter 2016
Linux internals for Database administrators at Linux Piter 2016
 
Autovacuum, explained for engineers, new improved version PGConf.eu 2015 Vienna
Autovacuum, explained for engineers, new improved version PGConf.eu 2015 ViennaAutovacuum, explained for engineers, new improved version PGConf.eu 2015 Vienna
Autovacuum, explained for engineers, new improved version PGConf.eu 2015 Vienna
 
PostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQPostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQ
 
Pgconfru 2015 kosmodemiansky
Pgconfru 2015 kosmodemianskyPgconfru 2015 kosmodemiansky
Pgconfru 2015 kosmodemiansky
 
Как PostgreSQL работает с диском
Как PostgreSQL работает с дискомКак PostgreSQL работает с диском
Как PostgreSQL работает с диском
 
Kosmodemiansky wr 2013
Kosmodemiansky wr 2013Kosmodemiansky wr 2013
Kosmodemiansky wr 2013
 
Максим Богук. Postgres-XC
Максим Богук. Postgres-XCМаксим Богук. Postgres-XC
Максим Богук. Postgres-XC
 
Иван Фролков. Tricky SQL
Иван Фролков. Tricky SQLИван Фролков. Tricky SQL
Иван Фролков. Tricky SQL
 
Илья Космодемьянский. Использование очередей асинхронных сообщений с PostgreSQL
Илья Космодемьянский. Использование очередей асинхронных сообщений с PostgreSQLИлья Космодемьянский. Использование очередей асинхронных сообщений с PostgreSQL
Илья Космодемьянский. Использование очередей асинхронных сообщений с PostgreSQL
 

Recently uploaded

Prach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism CommunityPrach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism Communityprachaibot
 
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptxTriangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptxRomil Mishra
 
multiple access in wireless communication
multiple access in wireless communicationmultiple access in wireless communication
multiple access in wireless communicationpanditadesh123
 
Levelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument methodLevelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument methodManicka Mamallan Andavar
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSsandhya757531
 
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONTHE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONjhunlian
 
CS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfCS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfBalamuruganV28
 
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Erbil Polytechnic University
 
Input Output Management in Operating System
Input Output Management in Operating SystemInput Output Management in Operating System
Input Output Management in Operating SystemRashmi Bhat
 
Forming section troubleshooting checklist for improving wire life (1).ppt
Forming section troubleshooting checklist for improving wire life (1).pptForming section troubleshooting checklist for improving wire life (1).ppt
Forming section troubleshooting checklist for improving wire life (1).pptNoman khan
 
70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical trainingGladiatorsKasper
 
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmComputer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmDeepika Walanjkar
 
CME 397 - SURFACE ENGINEERING - UNIT 1 FULL NOTES
CME 397 - SURFACE ENGINEERING - UNIT 1 FULL NOTESCME 397 - SURFACE ENGINEERING - UNIT 1 FULL NOTES
CME 397 - SURFACE ENGINEERING - UNIT 1 FULL NOTESkarthi keyan
 
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdfDEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdfAkritiPradhan2
 
priority interrupt computer organization
priority interrupt computer organizationpriority interrupt computer organization
priority interrupt computer organizationchnrketan
 
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSneha Padhiar
 
Katarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School CourseKatarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School Coursebim.edu.pl
 
STATE TRANSITION DIAGRAM in psoc subject
STATE TRANSITION DIAGRAM in psoc subjectSTATE TRANSITION DIAGRAM in psoc subject
STATE TRANSITION DIAGRAM in psoc subjectGayathriM270621
 
KCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitosKCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitosVictor Morales
 
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...Stork
 

Recently uploaded (20)

Prach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism CommunityPrach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism Community
 
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptxTriangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptx
 
multiple access in wireless communication
multiple access in wireless communicationmultiple access in wireless communication
multiple access in wireless communication
 
Levelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument methodLevelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument method
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
 
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONTHE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
 
CS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfCS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdf
 
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
 
Input Output Management in Operating System
Input Output Management in Operating SystemInput Output Management in Operating System
Input Output Management in Operating System
 
Forming section troubleshooting checklist for improving wire life (1).ppt
Forming section troubleshooting checklist for improving wire life (1).pptForming section troubleshooting checklist for improving wire life (1).ppt
Forming section troubleshooting checklist for improving wire life (1).ppt
 
70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training
 
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmComputer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
 
CME 397 - SURFACE ENGINEERING - UNIT 1 FULL NOTES
CME 397 - SURFACE ENGINEERING - UNIT 1 FULL NOTESCME 397 - SURFACE ENGINEERING - UNIT 1 FULL NOTES
CME 397 - SURFACE ENGINEERING - UNIT 1 FULL NOTES
 
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdfDEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdf
 
priority interrupt computer organization
priority interrupt computer organizationpriority interrupt computer organization
priority interrupt computer organization
 
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
 
Katarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School CourseKatarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School Course
 
STATE TRANSITION DIAGRAM in psoc subject
STATE TRANSITION DIAGRAM in psoc subjectSTATE TRANSITION DIAGRAM in psoc subject
STATE TRANSITION DIAGRAM in psoc subject
 
KCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitosKCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitos
 
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
 

PostgreSQL worst practices, version FOSDEM PGDay 2017 by Ilya Kosmodemiansky

  • 1. PostgreSQL worst practices at FOSDEM PGDay Brussels 2017 Ilya Kosmodemiansky ik@postgresql-consulting.com
  • 2. Best practices are just boring • Never follow them, try worst practices • Only those practices can really help you to screw the things up most effectively • PostgreSQL consultants are nice people, so try to make them happy
  • 3. How it works? • I have a list, a little bit more than 100 worst practices • I do not make this stuff up, all of them are real-life examples • I reshuffle my list every time before presenting and extract some amount of examples • Well, there are some things, which I like more or less, so it is not a very honest shuffle
  • 4. 0. Do not use indexes (a test one!) • Basically, there is no difference between full table scan and index scan • You can check that. Just insert 10 rows into a test table on your test server and compare. • Nobody deals with more than 10 row tables in production!
  • 5. 1. Use ORM • All databases share the same syntax • You must write database-independent code • Are there any benefits, which are based on database specific features? • It always good to learn a new complicated technology
  • 6. 2. Move joins to your application • Just select * a couple of tables into the application written in your favorite programming language • Than join them at the application level
  • 7. 2. Move joins to your application • Just select * a couple of tables into the application written in your favorite programming language • Than join them at the application level • Now you only need to implement nested loop join, hash join and merge join as well as query optimizer and page cache
  • 8. 3. Be in trend, be schema-less • You do not need to design the schema • You need only one table, two columns: id bigserial and extra jsonb • JSONB datatype is pretty effective in PostgreSQL, you can search in it just like in a well-structured table • Even if you put a 100M of JSON in it • Even if you have 1000+ tps
  • 9. 4. Be agile, use EAV • You need only 3 tables: entity, attribute, value
  • 10. 4. Be agile, use EAV • You need only 3 tables: entity, attribute, value • At some point add the 4th: attribute_type
  • 11. 4. Be agile, use EAV • You need only 3 tables: entity, attribute, value • At some point add the 4th: attribute_type • Whet it starts to work slow, just call those four tables The Core and add 1000+ tables with denormalized data
  • 12. 4. Be agile, use EAV • You need only 3 tables: entity, attribute, value • At some point add the 4th: attribute_type • Whet it starts to work slow, just call those four tables The Core and add 1000+ tables with denormalized data • If it is not enough, you can always add value_version
  • 13. 5. Try to create as many indexes as you can • Indexes consume no disk space • Indexes consume no shared_bufers • There is no overhead on DML if one and every column in a table covered with bunch of indexes • Optimizer will definitely choose your index once you created it • Keep calm and create more indexes
  • 14. 6. Always keep all your time series data • Time series data like tables with logs or session history should be never deleted, aggregated or archived, you always need to keep it all
  • 15. 6. Always keep all your time series data • Time series data like tables with logs or session history should be never deleted, aggregated or archived, you always need to keep it all • You will always know where to check, if you run out of disk space
  • 16. 6. Always keep all your time series data • Time series data like tables with logs or session history should be never deleted, aggregated or archived, you always need to keep it all • You will always know where to check, if you run out of disk space • You can always call that Big Data
  • 17. 6. Always keep all your time series data • Time series data like tables with logs or session history should be never deleted, aggregated or archived, you always need to keep it all • You will always know where to check, if you run out of disk space • You can always call that Big Data • Solve the problem using partitioning... one partition for an hour or for a minute
  • 18. 7. Turn autovacuum off • It is quite auxiliary process, you can easily stop it • There is no problem at all to have 100Gb data in a database which is 1Tb in size • 2-3Tb RAM servers are cheap, IO is a fastest thing in modern computing • Besides of that, everyone likes BigData
  • 19. 8. Keep master and slave on different hardware • That will maximize the possibility of unsuccessful failover
  • 20. 8. Keep master and slave on different hardware • That will maximize the possibility of unsuccessful failover • To make things worser, you can change only slave-related parameters at slave, leaving defaults for shared_buffers etc.
  • 21. 9. Put a synchronous replica to remote DC • Indeed! That will maximize availability!
  • 22. 9. Put a synchronous replica to remote DC • Indeed! That will maximize availability! • Especially, if you put the replica to another continent
  • 23. 10. Reinvent Slony • If you need some data replication to another database, try to implement it from scratch
  • 24. 10. Reinvent Slony • If you need some data replication to another database, try to implement it from scratch • That allows you to run into all problems, PostgreSQL have had since introducing Slony
  • 25. 11. Use as many count(*) as you can • Figure 301083021830123921 is very informative for the end user • If it changes in a second to 30108302894839434020, it is still informative • select count(*) from sometable is a quite light-weighted query • Tuple estimation from pg_catalog can never be precise enough for you
  • 26. 12. Never use graphical monitoring • You do not need graphs • Because it is an easy task to guess what was happened yesterday at 2 a.m. using command line and grep only
  • 27. 13. Never use Foreign Keys (Use local produced instead!) • Consistency control at application level always works as expected • You will never get data inconsistency without constraints • Even if you already have a bullet proof framework to maintain consistency, could it be good enough reason to use it?
  • 28. 14. Always use text type for all columns • It is always fun to reimplement date or ip validation in your code • You will never mistakenly convert ”12-31-2015 03:01AM” to ”15:01 12 of undef 2015” using text fields
  • 29. 15. Always use improved ”PostgreSQL” • Postgres is not a perfect database and you are smart • All that annoying MVCC staff, 32 bit xid and autovacuum nightmare look like they look because hackers are oldschool and lazy • Hack it in a hard way, do not bother yourself with submitting your patch to the community, just put it into production • It is easy to maintain such production and keep it compatible with ”not perfect” PostgreSQL upcoming versions
  • 30. 16. Postgres likes long transactions • Always call external services from stored procedures (like sending emails)
  • 31. 16. Postgres likes long transactions • Always call external services from stored procedures (like sending emails) • Oh, it is arguable... It can be, if 100% of developers were familiar with word timeout
  • 32. 16. Postgres likes long transactions • Always call external services from stored procedures (like sending emails) • Oh, it is arguable... It can be, if 100% of developers were familiar with word timeout • Anyway, you can just start transaction and go away for weekend
  • 33. 17. Load your data to PostgreSQL in a smart manner • Write your own loader, 100 parallel threads minimum
  • 34. 17. Load your data to PostgreSQL in a smart manner • Write your own loader, 100 parallel threads minimum • Never use COPY - it is specially designed for the task
  • 35. 18. Even if you want to backup your database... • Use replication instead of backup
  • 36. 18. Even if you want to backup your database... • Use replication instead of backup • Use pg_dump instead of backup
  • 37. 18. Even if you want to backup your database... • Use replication instead of backup • Use pg_dump instead of backup • Write your own backup script
  • 38. 18. Even if you want to backup your database... • Use replication instead of backup • Use pg_dump instead of backup • Write your own backup script • As complicated as possible, combine all external tools you know
  • 39. 18. Even if you want to backup your database... • Use replication instead of backup • Use pg_dump instead of backup • Write your own backup script • As complicated as possible, combine all external tools you know • Never perform a test recovery
  • 40. Do not forget That was WORST practice talk
  • 41. Questions or ideas? Share your story! ik@postgresql-consulting.com (I’am preparing this talk to be open sourced)