"In this session, we’ll detail Red Hat Storage Server data replication strategies for both near replication (LAN) and far replication (over WAN), and explain how replication has evolved over the last few years. You’ll learn about: Past mechanisms. Near replication (client-side replication). Far replication using timestamps (xtime). Present mechanisms. Near replication (server side) built using quorum and journaling. Faster far replication using journaling. Unified replication. Replication using snapshots. Stripe replication using erasure coding."

1. RED HAT STORAGE SERVER
REPLICATION: PAST AND PRESENT
Jeff Darcy, Venky Shankar, Raghavan Pichai
GlusterFS/RHS Developers @ Red Hat

2. Talk Outline
Background
Local replication
Remote replication
Next steps
Questions

3. Background
Types of replication, goals, and challenges

4. Synchronous Replication
S
S
Y
Y
N
N
C
C
+ high consistency - network sensitive

5. Quorum Enforcement
Replica #1 Replica #2 Replica #3
Majority can write Minority can’t
There can only be one majority => no split brain

6. Synchronous Replication Data Flows
X
X
X
Y
Y
Y
Chain Fan Out
Client
Server
Server
Client
Server
Server

7. Fan Out Replication
Y
Y
Y Client
Server
Server
Split
Bandwidth
Wait for
Slowest

8. Chain Replication
X
X
X
Client
Server
Server
Full
Bandwidth
Two Hops

9. Asynchronous Replication
A
A
S
S
C
C
Y
Y
N
N
+ low consistency - network insensitive

10. Effect of Network Partitions
A
A
S
S
MY
Y N
What’s the correct value?

11. Tradeoff Space
Network Sensitive Network Insensitive
High
Consistency
Low
Consistency
S
A

12. Red Hat Storage
Synchronous Near-Replication
Raghavan P
Developer, Red Hat

13. Traditional replication using AFR
“Automatic file replication”
Client based replication
Entry, meta data and data based replication.
Automated Self healing in case bricks recover after failure.

14. AFR Sequence Diagram
Client 1
Client 2
Server A
Server B
Lock
Pre Op
Op
Post Op
Unlock
Lock (blocked) Pre Op

15. AFR improvements
In 3.4 release
Eager locking
Piggybacking
Server quorum
In 3.5 release
Granular self heal
In 3.6 release
Rewrite of the code
Pending counters
Self healing in the context of self
heal daemon

16. NSR – new style (aka server side) replication
Replication to the back end (brick processes)
Controlled by a designated “leader” also known as sweeper.
Advantages
Bandwidth usage of client network optimized for direct (fuse) mounts
Avoidance of split brain
Sweeper elected using majority principle.
Per term Changelog on the sweeper preseves the ordering of operations.
Variable consistency models
for trading consistency with performance.

17. NSR high level blocks
NSR client side translator
Sends IO to sweeper
Sweeper (leader)
Forwards IO to peers
Commits after all peer completion
Non sweeper (follower)
Accepts IO only from sweeper or reconciliation
Rejects IO from client (client retry)
Change log
Reconciliation
Makes use of membership to figure out terms missing.
Makes use of change logs for syncing the corresponding terms.

18. NSR Sequence Diagram
Client 1
Client 2
Sweeper
Follower
Client 1 Request
Client 2 Request

19. Red Hat Storage Server
Geo-Replication
Venky Shankar
Developer, Red Hat

20. Geo-Replication
Asynchronous data replication
Continuous, Incremental
Across geographies
One site (master) to another (slave)
Multi-slave
Cascading
Fan-out
Disaster Recovery

21. Remote Replication: Past

22. Single node
Change detection
Crawling (xtime based crawl)
Data synchronization
Rsync
Suboptimal processing
rename, deletes, hardlink
Overview

23. Crawling and xtime
Xtime
Inode changed time
Marked up to root (marker xlator)
Crawling/Scanning
Directory crawl and file synchronization
xtime(master) > xtime(slave)
Slave xtime maintained by master

25. Remote Replication: Present

26. Overview
Multi node
Distributed (parallel) synchronization
Replica failover
Change detection
Consumable journals
Data synchronization (configurable)
Rsync, tar+ssh (large number of small files)
Efficient processing
rename, delete, hardlink

27. Journaling
Journaling Translator (changelog)
Records FOP (efficiently) local to a brick
Data, Entry, Metadata
Change detection : O(1) relative to number of changes
Consumer library (libgfchangelog)
Per brick
Publish/Subscribe mechanism
Journals periodically published

28. Remote Replication: Future

29. Replicating Snapshots
Multi Master
Vector clocks
Conflict detection & resolution
Libgfapi integration
Geo-replication to Swift target
Features

30. Red Hat Storage Server
Replication-related Features
Jeff Darcy
Developer, Red Hat

31. Unified Replication
Leader
Change
Log
Local
Replica
Change
Log
Remote
Replica
Change
LogSync Async

32. Erasure Coding (a.k.a. “disperse”)
D1 D2 D3 D4 P1 P2 P3
D1 D2 D3 D4 P1 P2 P3
D2

33. Also…
Volume
Snapshot
File
Snapshot
Deduplication
+
Compression
Checksums
OK
OK

34. Tiering (a.k.a. data classification)
Tier 0
Tier 1
Tier 2
SSD, no replication
Normal disk, sync replication
SMR disk, erasure coding
compression + checksums
async replication

35. Questions?