Webinar: PostgreSQL continuous backup and PITR with Barman

Twitter: @_GBartolini_ @2ndQuad https://www.2ndQuadrant.com
PostgreSQL continuous backup
and PITR with Barman - Webinar
How can I be ready for the
next recovery of a
PostgreSQL database?

PostgreSQL continuous
backup and PITR with
Barman
Webinar - 10 June 2020
Gabriele Bartolini
Head of Global Support

Something about me
● PostgreSQL user since ~2000
● Co-Founder of 2ndQuadrant
○ Head of Global Support
○ Leader of Cloud Native Initiative
○ Founding member of Barman
● Lean and DevOps practitioner
● Co-Founder of PostgreSQL
Europe and ITPUG

Public Asciinema screencasts
1. Verify PostgreSQL is up and running
2. Create 'barman' superuser and 'streaming_barman' replication user in
PostgreSQL
3. RPM installation overview
4. Our ﬁrst backup
5. Incremental backup based on hard links
6. Remote recovery and setup of Hot standby with on-demand WAL
fetching
7. Add WAL streaming to the standby server
8. Add 'zero data loss' or RPO=0 to the Primary/Standby/Barman cluster
9. Introduce geo-redundancy for a backed up server

Part One
Basic concepts for backup and recovery with
PostgreSQL

Logical backup with pg_dump
● Consistent snapshot of a PostgreSQL database
● Portable across major versions
● SQL format or Binary format
● Very useful
○ Development
○ Migrations
○ Dump and restore of portions of a database
● Not suitable for disaster recovery
○ Would you trust a home alarm system that takes a picture once a day?

Continuous physical backup
● Evolution of PostgreSQL’s crash recovery
○ Entire PostgreSQL instance, not a single database
● Founded on Write Ahead Logs
○ Also known as REDO log
○ Changes to the database are first written in the REDO logs, then in the data files
○ WAL files are by default 16MB
● Basic theory:
○ Archive WAL files on a different location
○ Take a physical copy of the PostgreSQL data files on a different location
● Introduced in PostgreSQL 8.1 (2005)

Continuous Recovery
● Founded on:
○ Standby servers
○ Continuous recovery
● Basic Theory:
○ Copy the data ﬁles on another server
○ Apply “redo” logs (roll forward)
■ Until the end of the WAL (full recovery)
■ Up to an exact moment (Point In Time Recovery)
○ Inspect or promote the server (exit continuous recovery)

WAL shipping
● Transfer WAL records to a different location
● WAL archiving:
○ Through PostgreSQL’s archive_command
■ Every time a WAL ﬁle is closed (16MB or archive_timeout)
○ Through native streaming replication
■ Continuously (WAL sender and WAL receiver)
■ Also synchronous (“zero data loss”)

Freedom & creativity
● Lego bricks
○ PostgreSQL continuous backup
○ WAL shipping
○ PostgreSQL continuous recovery
● Inﬁnite combinations
○ Architectures
○ Recovery objectives (RPO and RTO)
○ Processes
○ Use cases
● Temptation: DIY solution

Part Two
Key aspects of Barman

First backup & recovery tool for Postgres
● Conceived in 2011, Open sourced in 2012
● Linux & Python, started with Postgres 8.3
● Innovative concepts at that time:
○ Remote backup
○ Remote recovery
○ Multiple PostgreSQL servers
○ Backup catalogue and independent WAL archive
○ Monitoring integration
○ WAL compression
○ Retention policies
○ Incremental backup and recovery
○ Hook scripts

Requirements
1. Online backup
2. Automation
3. Management of frequency and retention
4. Off-site copies
5. Notiﬁcations of anomalies
6. Access limitations
7. Availability
8. Encryption

Lean approach (customers needs)
● Foster migrations from Oracle
● Parallel backup and recovery
○ Reduce backup and recovery times on very large databases
● Streaming replication support
○ Including synchronous replication for Zero Data loss backups
● On-Demand WAL fetching (WAL hub)
● Geo-redundancy
● Object stores interface (Barman Cloud)

Faster, Cheaper, Safer (DevOps)
● “We build it, we own it”
○ Barman team is on the front line of 2ndQ Support Incidents for Barman
● Multi-disciplinary development team
○ Peer review
● Trunk based development
● CI/CD and test automation
○ Build quality in / Shift left on security
○ Thousands of automated tests per commit
■ Unit tests, Smoke tests, Acceptance tests, Integration tests
■ All supported PostgreSQL versions and distributions
■ Release time went from 1 week to 2 hours

Part Three
Tips

#1 - Separate Barman from Postgres
● Let Barman and Postgres:
○ Run on different servers
○ Use different storage
● Think about:
○ Resilience
○ Commodity storage
■ Physical servers with high capacity local disks
○ Integration with standby servers
■ Zero data loss clusters

#2 - Avoid network disks
● … if you can
● Some customers run Barman on NFS/CIFS/…
● Think about:
○ File lock issues
■ Place “barman_lock_directory” in the local Linux disk
○ Performance issues
■ Especially if storage is shared

#3 - Favour locality over distance
● Let Barman and Postgres:
○ Be next to each other
○ In the same data centre
● Think about:
○ Data centre is a single point of failure - Yes, but don’t let it scare you
■ It can be solved

#4 - WAL streaming
● Let Postgres stream WAL records to Barman
○ Enable WAL streaming
○ Ensure all WALs are shipped
■ Rely on replication slots
■ Fully automated with “create_slot” option
● Think about:
○ Asynchronous replication by default
○ Synchronous replication if …
■ You add a standby server

#5 - WAL fetching
● Take advantage of WAL archive
● Use Barman as an “infinite” hub of WAL files
○ Forget about configuring “wal_keep_segments”
● On-demand remote fetching:
○ Standby servers
○ Recovery operations
● Parallel pre-fetch (performance boost)
● On the fly decompression
● “get-wal” feature

#6 - Don’t take yourself so seriously

#7 - Compress your WALs
● Set “compression” option, globally
● Performed by the WAL archiver process
○ Invoked automatically by the “cron” command
● gzip is normally ﬁne
● Supported algorithms:
○ bzip2
○ pigz

#8 - Your PostgreSQL building block
● The “Flux capacitor”:
○ Primary instance
○ Standby instance
○ Barman instance
● Think about:
○ High Availability
○ Disaster Recovery
○ Local standby servers
■ You can always add more
■ This architecture is simple yet very effective
○ Symmetric architecture (next rule)

#9 - Multi-data centre architecture

#10 - Encryption
● Encrypt at rest your:
○ PostgreSQL servers
○ Backup servers
● Think about:
○ Secure connections
○ Secure access to backup servers
○ Encryption of exported backup files:
■ Tar files on tape
■ Tar files on Cloud object stores

#11 - Deﬁne your maximum RPO
● Clearly deﬁne your Recovery Point Objective
○ For example maximum 5 minutes
● Set “archive_timeout”
○ For example “archive_timeout = 5min”

#12 - Reduce your RPO to 0
● Native synchronous replication
● Requirements:
○ A local standby
○ A local Barman
● Two options:
○ Zero data loss standby
○ Zero data loss backup

#13 - Backup from a standby
● Available with PostgreSQL 9.6
○ 9.2 to 9.5 requires the pgespresso extension
● Think about:
○ Symmetric architecture
○ Off-load the primary

#14 - Let “check” be your compass
● “barman check” is the most critical command
● It guides you
○ Setup process
○ Problem solving
● Think about:
○ Integration with alerting tools

#15 - Monitoring
● Barman is on a Linux system
● That system must be under monitoring
○ Standard metrics
○ Disk usage
● Think about:
○ “barman check --nagios all”

#16 - Weekly backups
● Start with weekly backups
○ Evaluate daily backups if you require shorter RTO
● Think about:
○ “barman backup all”

#17 - Retention policies
● Automatically purge old backups
● Retention policies based on:
○ Redundancy (quantity)
○ Recovery window (time, Point of Recoverability)
● Think about:
○ Delete hook scripts

#18 - Use rsync/SSH for backups
● Optimised algorithm
● Enables incremental copy
○ Set “reuse_backup = link” if your ﬁle system supports hard links
● Enables parallel copy
○ Set “parallel_jobs” option
● Think about:
○ Remember locality? Barman and Postgres are in the same network
■ Looser security measures

#19 - Rely on object stores
● By … relaying to AWS S3 compatible object stores
○ Requires Boto3 library
○ Use “barman-cloud-wal-archive” for WAL ﬁles
● Public cloud, Private cloud, Hybrid clouds
○ E.g. use a local object store that relays to public cloud
● Think about:
○ Removing single point of failure for data availability
■ Data centre, Provider, Continent, Planet, … that’s it for now
■ Multi-tiered backup and recovery
○ Enhanced disaster recovery capabilities
○ Remember encryption!

#20 - Public cloud, via Barman

#21 - Public cloud, direct

#22 - Local object store with gateway
To be implemented

#23 - Geo-redundancy

#24 - Aggressive start
● Enable “immediate_checkpoint”
● Speed up the start of the backup
○ Request a checkpoint without waiting for the scheduled one
● Think about:
○ Not available on standby servers

#25 - Precautions
● minimum_redundancy
○ Safety measure: set it to 1
● last_backup_maximum_age
○ Based on the backup frequency
● max_incoming_wals_queue
○ Let “check” fail if your incoming queue gets too high

#26 - Server configuration files
● Use a separate configuration file per server
● Place them in the /etc/barman.conf.d folder
● Suggested convention:
○ SERVER_ID-PGVERSION.conf
○ Example: juventus-12.conf

#27 - Last version always wins
● Always install the latest version of Barman
○ Trunk based development
○ Backwards compatible
● Test our development snapshots

#28 - Use our public repositories
● rpm.2ndquadrant.com
● apt.2ndquadrant.com
● Entry points:
○ Release level (default)
○ Snapshot level

#29 - Cross data centre backups
● If you have to …
● Use network compression
● Think about:
○ Symmetric architecture
○ Locality

#30 - Enjoy convention over configuration
● Most options in Barman:
○ Can be set globally in the configuration file
○ Can be overridden at server level
○ Have default values
● If you use our packages:
○ System configuration is already taken care of
■ User
■ Cron
■ Log rotation

#31 - Get hooked
● Hook scripts
● Before/After certain events
● Two types:
○ Standard: in case of failure, no retry
○ Retry: in case of failure, retry
■ Typical: before WAL archive to relay WAL ﬁles in the Cloud
● See “barman-cloud-wal-archive” man page

#32 - Work in small batches
● Conﬁgure archiver batch size
○ For standard archiving or streaming
○ By default unlimited batch
● Archiver process run by the “cron” command
○ Every minute
○ Tune it based on the number of expected WALs between two cron runs
○ Good value to start with is between 10 and 100

#33 - JSON output
● Every command supports “-f json”
● Integration with other applications

#34 - Principle Of Least Authority (POLA)
● Avoid using a superuser for Barman
● The “barman” user can be a standard user
○ With speciﬁc grants for backup and read operations
○ barman_streaming can be used for replication connections
● Requirements:
○ PostgreSQL 10+
○ Barman 2.11

#35 - Enhance gradually
● Consider:
○ One PostgreSQL instance
○ One Barman instance
● Think about:
○ Can I achieve Disaster Recovery?
○ Can I achieve High Availability?
○ Can I achieve Business Continuity?
○ What are my RPO and RTO?
○ Start with your goals, add components gradually to improve them

Part Four
What lies ahead of us

Version 2.11 (June 2020)
● “barman-cli-cloud” package
○ barman-cloud-wal-archive (experimental in 2.10)
○ barman-cloud-backup (experimental in 2.10)
○ barman-cloud-wal-restore
○ barman-cloud-restore
○ barman-cloud-list-backup
● Non superuser connections
● Bug ﬁxes

Tar format for backups
● Tier 2 - optional
● Introduce “tar_retention_policy”
○ >= retention_policy
● Compressed backups
● Encrypted backups

Object store relay
● Tier 3 - optional
● Introduce “s3_retention_policy”
○ >= tar_retention_policy
● Compressed backups
● Encrypted backups

Kubernetes
● Cloud Native PostgreSQL
○ Private operator by 2ndQuadrant
○ Public Docker images
● 1st stage:
○ Object store integration with barman-cli-cloud
○ Direct usage from the PostgreSQL primary node
■ Backup and WAL archive
● 2nd stage
○ Barman operator (Barman inside Kubernetes)

Part Five
Conclusions

Base backup information:
Disk usage : 24.2 TiB (24.2 TiB with WALs)
Incremental size : 6.7 TiB (-72.44%)
Timeline : 9
Begin WAL : 000000090000A9690000006A
End WAL : 000000090000A96D000000DC
WAL number : 1139
WAL compression ratio: 77.05%
Begin time : 2020-06-05 13:00:04.930924+00:00
End time : 2020-06-06 01:55:34.543064+00:00
Copy time : 12 hours, 47 minutes, 33 seconds + 42 seconds startup
Estimated throughput : 152.1 MiB/s ( 4 jobs)
Begin Offset : 13931024
End Offset : 14041528
Begin LSN : A969/6AD49210
End LSN : A96D/DCD641B8
WAL information:
No of files : 7136
Disk usage : 25.3 GiB
WAL rate : 90.79/hour
Compression ratio : 77.26%
Last available : 000000090000A989000000BC
Output of “barman show-backup” - Courtesy of Barman’s founding sponsor “Navionics”

Recap
● “Flux capacitor”
● RPO=0 scenarios
● Very Large Databases
● Stable, robust and backwards compatible
● Supports all supported PostgreSQL versions
● Used in mission critical environments
● Used in our infrastructure at 2ndQuadrant
○ We can say: “It saved us once”
● Enhance gradually
● Open source

License
Attribution 4.0 International (CC BY 4.0)
You are free to:
Share — copy and redistribute the material in any medium or
format
Adapt — remix, transform, and build upon the material for any
purpose, even commercially.
The licensor cannot revoke these freedoms as long as you
follow the license terms.

Questions?
www.pgbarman.org
info@2ndquadrant.com

Webinar: PostgreSQL continuous backup and PITR with Barman

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Webinar: PostgreSQL continuous backup and PITR with Barman

Similar to Webinar: PostgreSQL continuous backup and PITR with Barman (20)

More from Gabriele Bartolini

More from Gabriele Bartolini (8)

Recently uploaded

Recently uploaded (20)

Webinar: PostgreSQL continuous backup and PITR with Barman