Speaker: Kevin Albertson, Software Engineer II, MongoDB

1. Using Change Streams to
Keep Up With Your Data
Kevin Albertson, Software Engineer II

2. Kevin Albertson
Software Engineer II, MongoDB
@kevinAlbs

3. Why Real-Time Data?

4. March 8th, 2018

5. March 8th, 2018
Point A
Point B

8. March 8th, 2018
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:00
Stillw
aiting
7:15
Stillw
aiting

9. March 8th, 2018
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:15
Stillw
aiting
7:00
Stillw
aiting

10. 6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:30
Stillw
aiting
7:15
Stillw
aiting
7:00
Stillw
aiting

11. 6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:30
Stillw
aiting
7:15
Stillw
aiting
7:00
Stillw
aiting
7:45
Stillw
aiting

12. March 8th, 2018
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
8:00
A
nnouncem
ent
7:30
Stillw
aiting
7:15
Stillw
aiting
7:00
Stillw
aiting
7:45
Stillw
aiting

13. March 8th, 2018

14. Unnecessary
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
8:00
A
nnouncem
ent
7:30
Stillw
aiting
7:15
Stillw
aiting
7:00
Train
passes
7:45
Stillw
aiting
Arrive
in
N
YC
9:30
Take
train
in
otherdirection
9:30
Take
train
to
N
YC 11:00

15. Real-time is an Expectation

16. Change Streams Enable Existing Apps to
Provide Real-time Data

17. const changeStream =
db.collection('train').watch();
c h a n g e S t r e a m . o n ( ' c h a n g e ' , ( c h a n g e ) = > { c o n s o l e . l o g ( c h a n g e ) } ) ;
Added in 3.6
Improved in 4.0
{
"_id" : (resumeToken),
"operationType" : "insert",
"ns" : { "db" : "test", "coll" : "train" },
"documentKey" : {
"_id" : 123
},
"fullDocument" : {
"_id" : 123,
"text": "hello"
},
}

18. Change Stream
Characteristics

19. 1. Use Collection Access Controls
2. Present a Defined API
3. Scale Across Nodes
Change Streams

20. Return 5 Operation Types
1. Insert
2. Update
3. Replace
4. Delete
5. Invalidate

21. mongos
3 2
1
Shard 1 Shard 2 Shard 3
Total Ordering of Changes Across Shards

22. Documents Uniquely Identified
Shard key combined with _id to uniquely identify documents
mongos
Shard 2 Shard 3Shard 1

23. Changes are Durable
S
S
collection.watch()
P

24. Change Streams are Resumable
collection.watch()
PP
{
_id: <resumeToken>,
operationType: 'update'
...
}
SP
S

25. Change Streams are Resumable
changeStream.on ('change', (change) => {
console.log(change);
cachedResumeToken = change["_id"];
});
changeStream.on ('error', () => {
if (cachedResumeToken) {
establishChangeStream (cachedResumeToken);
}
});

26. Change Streams Utilize the Power of the
Aggregation Framework
$match $project $addFields $replaceRoot $redact
var changeStream = coll.watch([{ $match: {operationType: {$in: ['delete', 'replace']}}}]);

27. 1. Collection Access Controls
2. Defined API
3. Enable Scaling
4. Total Ordering
5. Durable
6. Resumable
7. Power of Aggregation

28. Types of Changes

29. Insert
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.insert({
_id: 1,
text: "hello"
})
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "insert",
"ns" : {
"db" : "test",
"coll" : "test"
},
"documentKey" : {
"_id" : 1
},
"fullDocument" : {
"_id" : 1,
"text": "hello"
},
}

30. Update
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.updateOne({_id: 1},
{ $set: {“text”: “updated”} })
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "update",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
},
"updateDescription" : {
"updatedFields" : {
"text" : "updated"
},
"removedFields" : [ ]
}
}

31. Update with fullDocument: updateLookup
> db.collection.watch([],
{ fullDocument: “updateLookup”,
maxAwaitTimeMS: 30000 }).pretty()
> db.collection.updateOne({_id: 1},
{ $set: {“number”: “5”} })
Shell 1 Shell 2
{ "_id" : (resumeToken),
"operationType" : "update",
"fullDocument" : {
"_id" : 1,
"text" : "updated",
"number" : 5
},
"ns": {"db": "test", "coll": "collection"},
"documentKey" : { "_id" : 1 },
"updateDescription" : {
"updatedFields" : { "number" : 5 },
"removedFields" : [ ]
}
}

32. Replace
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.replaceOne({_id: 1},
{text: "replaced"})
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "replace",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
},
"fullDocument" : {
"_id" : 1,
"text" : "replaced"
}
}

33. Delete
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.remove({_id: 1})
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "delete",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
}
}

34. Invalidate
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.drop()
Shell 1 Shell 2
{
"_id" : (token),
"operationType" : "invalidate"
}

35. New in 4.0

36. 1. startAtOperationTime
2. Change stream on db or client
3. Resumable before first change

37. Use Case

38. Real-time Train Arrival Estimates

39. Collection
> db.train.find().pretty()
{
"_id" : 11,
"position" : {
"lat" : 40.04714,
"lon" : -74.0351
}
}

40. App Caches State of All Trains
App Cache

41. _id: 11 (40.0, -74.0)
db.train.find().readConcern("majority")
App Cache
Populate the Cache

42. _id: 11 (40.0, -74.0)
db.train.watch([])
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: { lat: 40.41169 }
},
documentKey: {_id: 11}
Handled
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: {
passengerCount: 5
}
}
documentKey: {_id: 11}
Handled?
App Cache _id: 11 (40.4, -74.0)
Watch for changes

43. $match to Filter Irrelevant Changes

44. db.train.watch([{
$match: {
$or: [
{'updateDescription.updatedFields.lat': {$exists: 1}},
{'updateDescription.updatedFields.lon': {$exists: 1}}
]
}
}])
_id: (resumeToken),
operationType: 'insert',
fullDocument: {
_id: 16,
lat: 40.5889, lon: -76.1938
},
documentKey: {_id: 16}
Ignored?
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: {
passengerCount: 5
}
}
documentKey: {_id: 11}
Ignored
App Cache _id: 11 (40.4, -74.0)

45. Handle All Document Changes
If Caching State of Documents:

46. db.train.watch([{
$match: {
$or: [
{
operationType: 'update',
$or: [
{'updateDescription.updatedFields.lat': {$exists: 1}},
{'updateDescription.updatedFields.lon': {$exists: 1}}
]
},
{ operationType: { $in: ['delete', 'insert', 'replace'] } }
]
}
}])
App Cache _id: 11 (40.4, -74.0)
_id: (resumeToken),
operationType: 'insert',
fullDocument: {
_id: 16,
lat: 40.5889, lon: -76.1938
},
documentKey: {_id: 16}
Handled
_id: 16 (40.4, -74.0)
_id: (resumeToken),
operationType: 'delete',
documentKey: {_id: 11}Handled

47. Race Between Populating and Listening!
db.train.find().readConcern("majority")
.
.
.
db.train.watch()
update occurs!

48. find
Client Server
[ {_id: 11}…], $clusterTime: X
watch, startAtOperationTime: X
start session
startAtOperationTime

49. startAtOperationTime
const session = client.startSession();
const cursor = db.collection('train').find({}, { session: session });
populateApplicationCache(cursor);
const changeStream = db.collection('train').watch([], {
session: session,
startAtOperationTime: session.operationTime
});
changeStream.on('change', updateApplicationCache);

50. App Complete!
1. Doesn't miss updates
2. Handles network blips
3. Reduces bandwidth
4. Easy to prototype