1. #MongoDBDays - @m4rcsch
Replication and Replica Sets
Marc Schwering
Solutions Architect, 10gen

2. Notes to the presenter
Themes for this presentation:
• Balance between cost and redundancy.
• Cover the many scenarios which replication
would solve and why.
• Secret sauce of avoiding downtime & data loss
• If time is short, skip 'Behind the curtain' section
• If time is very short, skip Operational
Considerations also

3. Agenda
• Replica Sets Lifecycle
• Developing with Replica Sets
• Operational Considerations

4. Why Replication?
• How many have faced node failures?
• How many have been woken up from sleep to do
a fail-over(s)?
• How many have experienced issues due to
network latency?
• Different uses for data
– Normal processing
– Simple analytics

5. ReplicaSet Lifecycle

6. Node 1 Node 2
Node 3
Replica Set – Creation

7. Node 1 Node 2
Secondary Secondary
Heartbeat
Re
n
tio
pli
ica
cat
pl
on i
Re
Node 3
Primary
Replica Set – Initialize

8. Primary Election
Node 1 Node 2
Secondary Heartbeat Secondary
Node 3
Replica Set – Failure

9. Replication
Node 1 Node 2
Secondary Primary
Heartbeat
Node 3
Replica Set – Failover

10. Replication
Node 1 Node 2
Secondary Primary
Heartbeat
n
tio
ica
pl
Re
Node 3
Recovery
Replica Set – Recovery

11. Replication
Node 1 Node 2
Secondary Primary
Heartbeat
n
tio
ica
pl
Re
Node 3
Secondary
Replica Set – Recovered

12. ReplicaSet Roles &
Configuration

13. Node 1 Node 2
Secondary Arbiter
Heartbeat
Re
pli
cat
on i
Node 3
Primary
Replica Set Roles

14. Configuration Options
> conf = {
_id : "mySet",
members : [
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2},
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay :
3600}
]
}
> rs.initiate(conf)

15. Configuration Options
> conf = {
_id : "mySet”,
members : [ Primary DC
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2},
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay :
3600}
]
}
> rs.initiate(conf)

16. Configuration Options
> conf = {
Secondary DC
_id : "mySet”, Default Priority = 1
members : [
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2},
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay :
3600}
]
}
> rs.initiate(conf)

17. Configuration Options
> conf = {
_id : "mySet”,
members : [
{_id : 0, host : "A”, priority : 3}, Analytics
{_id : 1, host : "B", priority : 2}, node
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay :
3600}
]
}
> rs.initiate(conf)

18. Configuration Options
> conf = {
_id : "mySet”,
members : [
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2},
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay :
3600}
]
}
Backup node
> rs.initiate(conf)

19. Developing with Replica Sets

20. Client Application
Driver
Write
Read
Primary
Secondary Secondary
Strong Consistency

21. Client Application
Driver
Write
ad
Re
Re
ad
Primary
Secondary Secondary
Delayed Consistency

22. Write Concern
• Network acknowledgement
• Wait for error
• Wait for journal sync
• Wait for replication

23. Driver
write
Primary
apply in
memory
Unacknowledged

24. Driver
getLastError
Primary
apply in
memory
MongoDB Acknowledged (wait for error)

25. Driver
getLastError
j:true
write
Primary
apply in write to
memory journal
Wait for Journal Sync

26. Driver
getLastError
write
w:2
Primary
replicate
apply in
memory
Secondary
Wait for Replication

27. Tagging
• Control where data is written to, and read from
• Each member can have one or more tags
– tags: {dc: "ny"}
– tags: {dc: "ny",
subnet: "192.168",
rack: "row3rk7"}
• Replica set defines rules for write concerns
• Rules can change without changing app code

28. Tagging Example
{
_id : "mySet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}],
settings : {
getLastErrorModes : {
allDCs : {"dc" : 3},
someDCs : {"dc" : 2}} }
}
> db.blogs.insert({...})
> db.runCommand({getLastError : 1, w : "someDCs"})

29. Driver
getLastError
W:allDCs
write
Primary (SF)
replicate
apply in
memory
Secondary (NY)
replicate
Secondary (Cloud)
Wait for Replication (Tagging)

30. Read Preference Modes
• 5 modes
– primary (only) - Default
– primaryPreferred
– secondary
– secondaryPreferred
– Nearest
When more than one node is possible, closest node is used for
reads (all modes but primary)

31. Tagged Read Preference
• Custom read preferences
• Control where you read from by (node) tags
– E.g. { "disk": "ssd", "use": "reporting" }
• Use in conjunction with standard read preferences
– Except primary

32. Operational Considerations

33. Maintenance and Upgrade
• No downtime
• Rolling upgrade/maintenance
– Start with Secondary
– Primary last

34. Replica Set – 1 Data Center
• Single datacenter
Datacenter
• Single switch & power
Member 1 • Points of failure:
– Power
Member 2 – Network
– Data center
Datacenter 2
Member 3 – Two node failure
• Automatic recovery of
single node crash

35. Replica Set – 2 Data Centers
• Multi data center
Datacenter 1
• DR node for safety
Member 1
• Can’t do multi data
Member 2 center durable write
safely since only 1 node
in distant DC
Datacenter 2
Member 3

36. Replica Set – 3 Data Centers
Datacenter 1 • Three data centers
Member 1
Member 2
• Can survive full data
center loss
Datacenter 2
Member 3 • Can do w= { dc : 2 } to
Member 4 guarantee write in 2
data centers (with tags)
Datacenter 3
Member 5

37. Behind the Curtain

38. Implementation details
• Heartbeat every 2 seconds
– Times out in 10 seconds
• Local DB (not replicated)
– system.replset
– oplog.rs
• Capped collection
• Idempotent version of operation stored

39. Op(erations) Log is idempotent
> db.replsettest.insert({_id:1,value:1})
{ "ts" : Timestamp(1350539727000, 1), "h" :
NumberLong("6375186941486301201"), "op" : "i", "ns" :
"test.replsettest", "o" : { "_id" : 1, "value" : 1 } }
> db.replsettest.update({_id:1},{$inc:{value:10}})
{ "ts" : Timestamp(1350539786000, 1), "h" :
NumberLong("5484673652472424968"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 1 },
"o" : { "$set" : { "value" : 11 } } }

40. Single operation can have many
entries
> db.replsettest.update({},{$set:{name : ”foo”}, false,
true})
{ "ts" : Timestamp(1350540395000, 1), "h" :
NumberLong("-4727576249368135876"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 2 }, "o" :
{ "$set" : { "name" : "foo" } } }
{ "ts" : Timestamp(1350540395000, 2), "h" :
NumberLong("-7292949613259260138"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 3 }, "o" :
{ "$set" : { "name" : "foo" } } }
{ "ts" : Timestamp(1350540395000, 3), "h" :
NumberLong("-1888768148831990635"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 1 }, "o" :
{ "$set" : { "name" : "foo" } } }

41. Recent improvements
• Read preference support with sharding
– Drivers too
• Improved replication over WAN/high-latency
networks
• rs.syncFrom command
• buildIndexes setting
• replIndexPrefetch setting

42. Just Use It
• Use replica sets
• Easy to setup
– Try on a single machine
• Check doc page for RS tutorials
– http://docs.mongodb.org/manual/replication/#tutorials

43. #MongoDBDays - @m4rcsch
Thank You
Marc Schwering
Solutions Architect, 10gen