Apache Kafka has shown that the log is a powerful abstraction for data-intensive applications. It can play a key role in managing data and distributing it across the enterprise efficiently. Vital to any data plane is not just performance, but availability and scalability. In this session, we examine what a distributed log is, how it works, and how it can achieve these goals. Specifically, we'll discuss lessons learned while building NATS Streaming, a reliable messaging layer built on NATS that provides similar semantics. We'll cover core components like leader election, data replication, log persistence, and message delivery. Come learn about distributed systems!

1. Building a Distributed
Message Log from
Scratch
Tyler Treat · Iowa Code Camp · 11/04/17

2. - Messaging Nerd @ Apcera
- Working on nats.io
- Distributed systems
- bravenewgeek.com
Tyler Treat

4. - The Log
-> What?
-> Why?
- Implementation
-> Storage mechanics
-> Data-replication techniques
-> Scaling message delivery
-> Trade-offs and lessons learned
Outline

5. The Log

6. The Log
A totally-ordered,
append-only data
structure.

7. The Log
0

8. 0 1
The Log

9. 0 1 2
The Log

10. 0 1 2 3
The Log

11. 0 1 2 3 4
The Log

12. 0 1 2 3 4 5
The Log

13. 0 1 2 3 4 5
newest recordoldest record
The Log

14. newest recordoldest record
The Log

15. Logs record what
happened and when.

16. caches
databases
indexes
writes

18. Examples in the wild:
-> Apache Kafka
-> Amazon Kinesis
-> NATS Streaming
-> Tank

19. Key Goals:
-> Performance
-> High Availability
-> Scalability

20. The purpose of this talk is to learn…
-> a bit about the internals of a log abstraction.
-> how it can achieve these goals.
-> some applied distributed systems theory.

21. You will probably never need to
build something like this yourself,
but it helps to know how it works.

22. Implemen-
tation

23. Implemen-
tation
Don’t try this at
home.

24. Some first principles…
Storage Mechanics
• The log is an ordered, immutable sequence of messages
• Messages are atomic (meaning they can’t be broken up)
• The log has a notion of message retention based on some policies
(time, number of messages, bytes, etc.)
• The log can be played back from any arbitrary position
• The log is stored on disk
• Sequential disk access is fast*
• OS page cache means sequential access often avoids disk

25. http://queue.acm.org/detail.cfm?id=1563874

26. avg-cpu: %user %nice %system %iowait %steal %idle
13.53 0.00 11.28 0.00 0.00 75.19
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvda 0.00 0.00 0.00 0 0
iostat

27. Storage Mechanics
log file
0

28. Storage Mechanics
log file
0 1

29. Storage Mechanics
log file
0 1 2

30. Storage Mechanics
log file
0 1 2 3

31. Storage Mechanics
log file
0 1 2 3 4

32. Storage Mechanics
log file
0 1 2 3 4 5

33. Storage Mechanics
log file
…
0 1 2 3 4 5

34. Storage Mechanics
log segment 3 filelog segment 0 file
0 1 2 3 4 5

35. Storage Mechanics
log segment 3 filelog segment 0 file
0 1 2 3 4 5
0 1 2 0 1 2
index segment 0 file index segment 3 file

36. Zero-copy Reads
user space
kernel space
page cache
disk
socket
NIC
application
read send

37. Zero-copy Reads
user space
kernel space
page cache
disk NIC
sendfile

38. Left as an exercise for the listener…
-> Batching
-> Compression

39. caches
databases
indexes
writes

40. caches
databases
indexes
writes

41. caches
databases
indexes
writes

42. How do we achieve high availability
and fault tolerance?

43. Questions:
-> How do we ensure continuity of reads/writes?
-> How do we replicate data?
-> How do we ensure replicas are consistent?
-> How do we keep things fast?
-> How do we ensure data is durable?

44. Questions:
-> How do we ensure continuity of reads/writes?
-> How do we replicate data?
-> How do we ensure replicas are consistent?
-> How do we keep things fast?
-> How do we ensure data is durable?

45. caches
databases
indexes
writes

46. Questions:
-> How do we ensure continuity of reads/writes?
-> How do we replicate data?
-> How do we ensure replicas are consistent?
-> How do we keep things fast?
-> How do we ensure data is durable?

47. Data-Replication Techniques
1. Gossip/multicast protocols
Epidemic broadcast trees, bimodal multicast, SWIM, HyParView, NeEM
2. Consensus protocols
2PC/3PC, Paxos, Raft, Zab, chain replication

48. Questions:
-> How do we ensure continuity of reads/writes?
-> How do we replicate data?
-> How do we ensure replicas are consistent?
-> How do we keep things fast?
-> How do we ensure data is durable?

49. Data-Replication Techniques
1. Gossip/multicast protocols
Epidemic broadcast trees, bimodal multicast, SWIM, HyParView, NeEM
2. Consensus protocols
2PC/3PC, Paxos, Raft, Zab, chain replication

50. Consensus-Based Replication
1. Designate a leader
2. Replicate by either:
a) waiting for all replicas
—or—
b) waiting for a quorum of replicas

51. Pros Cons
All Replicas
Tolerates f failures with
f+1 replicas
Latency pegged to
slowest replica
Quorum
Hides delay from a slow
replica
Tolerates f failures with
2f+1 replicas
Consensus-Based Replication

52. Replication in Kafka
1. Select a leader
2. Maintain in-sync replica set (ISR) (initially every replica)
3. Leader writes messages to write-ahead log (WAL)
4. Leader commits messages when all replicas in ISR ack
5. Leader maintains high-water mark (HW) of last
committed message
6. Piggyback HW on replica fetch responses which
replicas periodically checkpoint to disk

53. 0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
Replication in Kafka

54. Failure Modes
1. Leader fails

55. 0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
Leader fails

56. 0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
Leader fails

57. 0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
Leader fails

58. 0 1 2 3
HW: 3
0 1 2 3
HW: 3
b2 (leader)
b3 (follower)ISR: {b2, b3}
writes
Leader fails

59. Failure Modes
1. Leader fails
2. Follower fails

60. 0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
Follower fails

61. 0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
Follower fails

62. 0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
Follower fails
replica.lag.time.max.ms

63. 0 1 2 3 4 5
b1 (leader)
HW: 3
0 1 2 3
HW: 3
b3 (follower)ISR: {b1, b3}
writes
Follower fails
replica.lag.time.max.ms

64. Failure Modes
1. Leader fails
2. Follower fails
3. Follower temporarily partitioned

65. 0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
Follower temporarily
partitioned

66. Follower temporarily
partitioned
0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes

67. Follower temporarily
partitioned
0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
replica.lag.time.max.ms

68. Follower temporarily
partitioned
0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 3
0 1 2 3
HW: 3
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2}
writes
replica.lag.time.max.ms

69. Follower temporarily
partitioned
0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 5
0 1 2 3
HW: 5
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2}
writes
5

70. Follower temporarily
partitioned
0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 5
0 1 2 3
HW: 5
HW: 3
b2 (follower)
b3 (follower)ISR: {b1, b2}
writes
5

71. Follower temporarily
partitioned
0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 5
0 1 2 3
HW: 5
HW: 4
b2 (follower)
b3 (follower)ISR: {b1, b2}
writes
5
4

72. Follower temporarily
partitioned
0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 5
0 1 2 3
HW: 5
HW: 5
b2 (follower)
b3 (follower)ISR: {b1, b2}
writes
5
4 5

73. Follower temporarily
partitioned
0 1 2 3 4 5
b1 (leader)
0 1 2 3 4HW: 5
0 1 2 3
HW: 5
HW: 5
b2 (follower)
b3 (follower)ISR: {b1, b2, b3}
writes
5
4 5

74. Replication in NATS Streaming
1. Metadata Raft group replicates client state
2. Separate Raft group per topic replicates messages
and subscriptions
3. Conceptually, two logs: Raft log and message log

75. http://thesecretlivesofdata.com/raft

76. Challenges
1. Scaling Raft

77. Scaling Raft
With a single topic, one node is elected leader and it
heartbeats messages to followers

78. Scaling Raft
As the number of topics increases unbounded, so do the
number of Raft groups.

79. Scaling Raft
Technique 1: run a fixed number of Raft groups and use
a consistent hash to map a topic to a group.

80. Scaling Raft
Technique 2: run an entire node’s worth of topics as a
single group using a layer on top of Raft.
https://www.cockroachlabs.com/blog/scaling-raft

81. Challenges
1. Scaling Raft
2. Dual writes

82. Dual Writes
Raft
Store
committed

83. Dual Writes
msg 1Raft
Store
committed

84. Dual Writes
msg 1 msg 2Raft
Store
committed

85. Dual Writes
msg 1 msg 2Raft
msg 1 msg 2Store
committed

86. Dual Writes
msg 1 msg 2 subRaft
msg 1 msg 2Store
committed

87. Dual Writes
msg 1 msg 2 sub msg 3Raft
msg 1 msg 2Store
committed

88. Dual Writes
msg 1 msg 2 sub msg 3
add
peer
msg 4Raft
msg 1 msg 2 msg 3Store
committed

89. Dual Writes
msg 1 msg 2 sub msg 3
add
peer
msg 4Raft
msg 1 msg 2 msg 3Store
committed

90. Dual Writes
msg 1 msg 2 sub msg 3
add
peer
msg 4Raft
msg 1 msg 2 msg 3 msg 4Store
commit

91. Dual Writes
msg 1 msg 2 sub msg 3
add
peer
msg 4Raft
msg 1 msg 2 msg 3 msg 4Store
0 1 2 3 4 5
0 1 2 3
physical offset
logical offset

92. Dual Writes
msg 1 msg 2 sub msg 3
add
peer
msg 4Raft
msg 1 msg 2Index
0 1 2 3 4 5
0 1 2 3
physical offset
logical offset
msg 3 msg 4

93. Treat the Raft log as our message
write-ahead log.

94. Questions:
-> How do we ensure continuity of reads/writes?
-> How do we replicate data?
-> How do we ensure replicas are consistent?
-> How do we keep things fast?
-> How do we ensure data is durable?

95. Performance
1. Publisher acks
-> broker acks on commit (slow but safe)
-> broker acks on local log append (fast but unsafe)
-> publisher doesn’t wait for ack (fast but unsafe)
2. Don’t fsync, rely on replication for durability
3. Keep disk access sequential and maximize zero-copy reads
4. Batch aggressively

96. Questions:
-> How do we ensure continuity of reads/writes?
-> How do we replicate data?
-> How do we ensure replicas are consistent?
-> How do we keep things fast?
-> How do we ensure data is durable?

97. Durability
1. Quorum guarantees durability
-> Comes for free with Raft
-> In Kafka, need to configure min.insync.replicas and acks, e.g.
topic with replication factor 3, min.insync.replicas=2, and
acks=all
2. Disable unclean leader elections
3. At odds with availability,
i.e. no quorum == no reads/writes

98. Scaling Message Delivery
1. Partitioning

99. Partitioning is how we scale linearly.

100. caches
databases
indexes
writes

101. HELLA WRITES
caches
databases
indexes

102. caches
databases
indexes
HELLA WRITES

103. caches
databases
indexes
writes
writes
writes
writes
Topic: purchases
Topic: inventory

104. caches
databases
indexes
writes
writes
writes
writes
Topic: purchases
Topic: inventory
Accounts A-M
Accounts N-Z
SKUs A-M
SKUs N-Z

105. Scaling Message Delivery
1. Partitioning
2. High fan-out

106. High Fan-out
1. Observation: with an immutable log, there are no
stale/phantom reads
2. This should make it “easy” (in theory) to scale to a
large number of consumers (e.g. hundreds of
thousands of IoT/edge devices)
3. With Raft, we can use “non-voters” to act as read
replicas and load balance consumers

107. Scaling Message Delivery
1. Partitioning
2. High fan-out
3. Push vs. pull

108. Push vs. Pull
• In Kafka, consumers pull data from brokers
• In NATS Streaming, brokers push data to consumers
• Pros/cons to both:
-> With push we need flow control; implicit in pull
-> Need to make decisions about optimizing for
latency vs. throughput
-> Thick vs. thin client and API ergonomics

109. Scaling Message Delivery
1. Partitioning
2. High fan-out
3. Push vs. pull
4. Bookkeeping

110. Bookkeeping
• Two ways to track position in the log:
-> Have the server track it for consumers
-> Have consumers track it
• Trade-off between API simplicity and performance/server
complexity
• Also, consumers might not have stable storage (e.g. IoT device,
ephemeral container, etc.)
• Can we split the difference?

111. Offset Storage
• Can store offsets themselves in the log (in Kafka,
originally had to store them in ZooKeeper)
• Clients periodically checkpoint offset to log
• Use log compaction to retain only latest offsets
• On recovery, fetch latest offset from log

112. Offset Storage
bob-foo-0
11
alice-foo-0
15Offsets
0 1 2 3
bob-foo-1
20
bob-foo-0
18
4
bob-foo-0
21

113. Offset Storage
bob-foo-0
11
alice-foo-0
15Offsets
0 1 2 3
bob-foo-1
20
bob-foo-0
18
4
bob-foo-0
21

114. Offset Storage
alice-foo-0
15
bob-foo-1
20Offsets
1 2 4
bob-foo-0
21

115. Offset Storage
Advantages:
-> Fault-tolerant
-> Consistent reads
-> High write throughput (unlike ZooKeeper)
-> Reuses existing structures, so less server
complexity

116. Trade-offs and Lessons Learned
1. Competing goals

117. Competing Goals
1. Performance
-> Easy to make something fast that’s not fault-tolerant or scalable
-> Simplicity of mechanism makes this easier
-> Simplicity of “UX” makes this harder
2. Scalability (and fault-tolerance)
-> Scalability and FT are at odds with simplicity
-> Cannot be an afterthought—needs to be designed from day 1
3. Simplicity (“UX”)
-> Simplicity of mechanism shifts complexity elsewhere (e.g. client)
-> Easy to let server handle complexity; hard when that needs to be
distributed and consistent while still being fast

118. Trade-offs and Lessons Learned
1. Competing goals
2. Availability vs. Consistency

119. Availability vs. Consistency
• CAP theorem
• Consistency requires quorum which hinders
availability and performance
• Minimize what you need to replicate

120. Trade-offs and Lessons Learned
1. Competing goals
2. Availability vs. Consistency
3. Aim for simplicity

121. Distributed systems are complex enough.
Simple is usually better (and faster).

122. Trade-offs and Lessons Learned
1. Competing goals
2. Availability vs. Consistency
3. Aim for simplicity
4. Lean on existing work

123. Don’t roll your own coordination protocol,
use Raft, ZooKeeper, etc.

124. Trade-offs and Lessons Learned
1. Competing goals
2. Availability vs. Consistency
3. Aim for simplicity
4. Lean on existing work
5. There are probably edge cases for which you
haven’t written tests

125. There are many failure modes, and you can
only write so many tests.
Formal methods and property-based/
generative testing can help.

127. Trade-offs and Lessons Learned
1. Competing goals
2. Availability vs. Consistency
3. Aim for simplicity
4. Lean on existing work
5. There are probably edge cases for which you
haven’t written tests
6. Be honest with your users

128. Don’t try to be everything to everyone. Be
explicit about design decisions, trade-
offs, guarantees, defaults, etc.

129. Thanks!
@tyler_treat
bravenewgeek.com