This presentation introduces every aspects of the Kafka ecosystem: - Concepts: explain all misleading concepts such as topic vs partition vs replication, producer vs consumer vs consumer group, group leader vs group coordinator, ... - Advanced concepts: delivery semantic; idempotent producers; isolation levels; differences between the offsets such as High Watermark, Log End Offset, Last Stable Offset ... - Kafka architecture: explain all Kafka components such as brokers, controllers, zookeeper, ... - Overview of Kafka security: TLS/SSL, SASL, Kerberos, ... - Overview of Kafka ecosystem: Kafka Stream, Kafka Connect, Schema Registry, monitoring tools. - Kafka in Golang: How to use Kafka client in Golang. - Comparison with other message queues such as RabbitMQ.

1. Ka#ka
Huynh Quang Thao
Trusting Social

2. Message Queue
- Message queues provide an asynchronous communications protocol,
meaning that the sender and receiver of the message do not need to
interact with the message queue at the same time.
- Messages placed onto the queue are stored until the recipient
retrieves them. (Wikipedia)

3. Overview Architecture

4. Ka#ka Architecture - General

5. Ka#ka Architecture - Partition

6. Ka#ka Architecture - replication

7. Ka#ka Architecture - Consumer Group

8. Ka#ka Architecture - overview

9. Ka#ka Architecture

10. Ka#ka at Grab
https://engineering.grab.com/quotas-service

11. Ka#ka at Reddit
https://redditblog.com/2017/05/24/view-counting-at-reddit/

12. Ka#ka Concepts

13. Zookeeper
- It is essentially a centralized service for distributed systems to a
hierarchical key-value store, which is used to provide a distributed
con#iguration service, synchronization service, and naming registry
for large distributed systems. (Wikipedia)

14. Zookeeper
- It is essentially a centralized service for distributed systems to a
hierarchical key-value store, which is used to provide a distributed
con#iguration service, synchronization service, and naming registry
for large distributed systems. (Wikipedia)
- Offer high availability: Ka#ka, Spark, Solr, Hadoop.

15. Zookeeper
- It is essentially a centralized service for distributed systems to a
hierarchical key-value store, which is used to provide a distributed
con#iguration service, synchronization service, and naming registry
for large distributed systems. (Wikipedia)
- Offer high availability: Ka#ka, Spark, Solr, Hadoop.
- Other similar services: etcd (used in Kubernetes), consult.

16. Zookeeper
- It is essentially a centralized service for distributed systems to a
hierarchical key-value store, which is used to provide a distributed
con#iguration service, synchronization service, and naming registry
for large distributed systems. (Wikipedia)
- Offer high availability: Ka#ka, Spark, Solr, Hadoop.
- Other similar services: etcd (used in Kubernetes), consult.
- Ka#ka uses the Zookeeper to maintains broker con#igurations,
consumer con#igurations (old version), available broker nodes,
electing controller, service discovery, …

17. Broker & Controller
- A Ka#ka cluster consists of one or more servers which are called
brokers.
- Producers are processes that publish the data into Ka#ka topics which
stored in the brokers.

18. Broker & Controller
- In a Ka#ka cluster, one of the brokers serves as the controller.
- Elected using leader election on the Zookeeper.
- Responsible for managing the states of partitions and replicas.
- Electing partition leader.

19. Broker & Controller
- In a Ka#ka cluster, one of the brokers serves as the controller.
- Responsible for managing the states of partitions and replicas.
- Electing partition leader.
- Elected using leader election on the Zookeeper.

20. Topic-Partition-Replication
- The topic is the place where data is published by producers and is
pulled by consumers.

21. Topic-Partition-Replication
- The topic is the place where data is published by producers and is
pulled by consumers.
- The topic is divided into several partitions.
- Each partition is ordered and messages within a partition get an
incrementing id, called offset.
- Partitions allow us to parallelize the consuming work by splitting the
data into multiple places.

22. Topic-Partition-Replication
- The topic is the place where data is published by producers and is
pulled by consumers.
- The topic is divided into several partitions.
- Each partition is ordered and messages within a partition get an
incrementing id, called offset.
- Partitions allow us to parallelize the consuming work by splitting the
data into multiple places.
- Every topic partition in Ka:ka is replicated n times, which n is the
replication factor of the topic.
- Each replication will be on a separate broker.
- The replication factor must be less than or equal to the total brokers.

23. Topic-Partition-Replication
- The topic is the place where data is published by producers and is
pulled by consumers.
- The topic is divided into several partitions.
- Each partition is ordered and messages within a partition get an
incrementing id, called offset.
- Partitions allow us to parallelize the consuming work by splitting the
data into multiple places.
- Every topic partition in Ka:ka is replicated n times, which n is the
replication factor of the topic.
- Each replication will be on a separate broker.
- The replication factor must be less than or equal to the total brokers.
Question: Can the consumer read data from the replica?

24. Topic-Partition-Replication
- Order is guaranteed only within the partition.
- Data is kept only for a limited time. (default 2 weeks)
- Once data is written to a partition, it can’t be changed.
- Data is assigned randomly to a partition unless a key is provided.

25. Topic-Partition-Replication

26. Partition Leader/Follower/ISR
- Ka#ka chooses one partition’s replica as the leader and all other
partition’s leaders as followers.
- All producers will write to this leader and all consumers will read from
this leader.
- A follower that is in-sync called an ISR. (in-sync replica)
- Once the leader is down, Ka#ka will elect one follower in the "ISR set" to
become the leader.

27. Partition Leader/Follower/ISR
- Ka#ka chooses one partition’s replica as the leader and all other
partition’s leaders as followers.
- All producers will write to this leader and all consumers will read from
this leader.
- A follower that is in-sync called an ISR. (in-sync replica)
- Once the leader is down, Ka#ka will elect one follower in the "ISR set" to
become the leader.
Question: How does the Ka#ka maintain the ISR set for each partition?

28. Producer and Consumer
- Producer: a Ka#ka client that publishes messages to the Ka#ka cluster.
- Consumer: a Ka#ka client that reads messages which is published by
publishers at their own pace.

29. Producer
- The producer has to specify the topic name and at least one broker to
connect.
- It it better to specify multiple brokers for the high availability.
- Ka#ka will automatically take care of routing the data to the right
brokers.

30. Producer - acknowledgement
Producers can set the acknowledgment con#iguration for sending data.
Acks = 0
- The message is pushed to the socket buffer.
- The producer won't wait for the acknowledgment from the leader.
Acks = 1
- The leader will append the message to its log and then returns the
acknowledgment to the producer .
- The producer will wait for the acknowledgment from the leader.
Acks = all
- The leader will append the message to its log.
- The leader will wait for all acknowledgments from all in-sync replicas.
- The producer will wait for the leader's acknowledgment.
- No data loss.

31. Producer - acknowledgement
Producers can set the acknowledgment con#iguration for sending data.
Acks = 0
- The message is pushed to the socket buffer.
- The producer won't wait for the acknowledgment from the leader.
Acks = 1
- The leader will append the message to its log and then returns the
acknowledgment to the producer .
- The producer will wait for the acknowledgment from the leader.
Acks = all
- The leader will append the message to its log.
- The leader will wait for all acknowledgments from all in-sync replicas.
- The producer will wait for the leader's acknowledgment.
- No data loss.
Question: How does choosing the ACK value affect the in-sync replica set?

32. Producer - key message
- Producers can choose to send a key with the message.
- If a key is sent along with the message, it guarantees that all messages
with the same key will always store in the same partition.
- Some Ka#ka functionalities based on the key message: log compaction,
cleaning up offsets ...

33. Consumer
- The consumer has to specify the topic name, partition (optional) and at
least one broker to connect.
- It it better to specify multiple brokers for the high availability.
- Ka#ka will automatically take care of pulling data from the right brokers.
- Data is read in order in each partition.

34. Consumer - Offset types
High Watermark:
- The offset of messages that are fully replicated to all ISR-replicas.
Log End Offset:
- The latest offset of messages on the leader partition.
- Consumer only reads up to the high watermark.
Leader Partition
1
2
3
4 (HW)
5
6
7 (LEO)
ISR-Replica 1
1
2
3
4
5
6
7
ISR-Replica 2
1
2
3
4
Out-of-sync
1
2
replica.lag.max.messages=4

35. Consumer - Offset types
High Watermark:
- The offset of messages that are fully replicated to all ISR-replicas.
Log End Offset:
- The latest offset of messages on the leader partition.
- Consumer only reads up to the high watermark.
Leader Partition
1
2
3
4
5
6
7 (HW + LEO)
ISR-Replica 1
1
2
3
4
5
6
7
ISR-Replica 2
1
2
3
4
5
6
7
Out-of-sync
1
2
replica.lag.max.messages=4

36. Consumer Group
- Messaging traditionally has two models: queuing and publish-
subscribe.
- queuing: allows us to divide up the processing of data over multiple
consumer instances, which lets you scale your processing
- Publish-subscribe: allow us broadcast data to multiple processes but
has no way of scaling processing since every message goes to every
subscriber.
- The consumer group concept in Ka#ka generalizes these two concepts.

37. Consumer Group
- Consumers can join a group by using the same group.id.
- Ka#ka assigns partitions to the consumers in the same group.
- Each partition is consumed by exactly one consumer in the group.

38. Consumer Group
- Consumers can join a group by using the same group.id.
- Ka#ka assigns partitions to the consumers in the same group.
- Each partition is consumed by exactly one consumer in the group.
- Each consumer within a group reads from exclusive partitions.
- One consumer can consume multiple partitions.
- It cannot have more consumers than partitions. Otherwise, some
consumers will be inactive state.

39. Consumer Group
- Consumers can join a group by using the same group.id.
- Ka#ka assigns partitions to the consumers in the same group.
- Each partition is consumed by exactly one consumer in the group.
- Each consumer within a group reads from exclusive partitions.
- One consumer can consume multiple partitions.
- It cannot have more consumers than partitions. Otherwise, some
consumers will be inactive state.
Question: Do Ka#ka allow one topic have multiple consumer groups?

40. Consumer Group
- Consumers can join a group by using the same group.id.
- Ka#ka assigns partitions to the consumers in the same group.
- Each partition is consumed by exactly one consumer in the group.
- Each consumer within a group reads from exclusive partitions.
- One consumer can consume multiple partitions.
- It cannot have more consumers than partitions. Otherwise, some
consumers will be inactive state.
Question: Do Ka#ka allow one topic have multiple consumer groups?
Answer: Yes
- E.g: consumer groups for saving the data to database. Consumer group
for transforming the data into other systems.

41. Consumer Group

42. Consumer Group

43. Consumer Group

44. Consumer Offset
- Ka#ka stores the consumer offset in the topic named "_
_consumer_offsets"
- [Group, Topic, Partition] -> [Offset, MetaData, TimeStamp]
- Automatically create when a consumer using a group connects to the
cluster.
- This information stored in the Ka#ka cluster (old version: Zookeeper).
- The consumer should commit the offset automatically or manually after
reading the message.

45. Consumer Offset
- Ka#ka stores the consumer offset in the topic named "_
_consumer_offsets"
- [Group, Topic, Partition] -> [Offset, MetaData, TimeStamp]
- Automatically create when a consumer using a group connects to the
cluster.
- This information stored in the Ka#ka cluster (old version: Zookeeper).
- The consumer should commit the offset automatically or manually after
reading the message.
Question: What if the "_ _consumer_offsets" topic grows huge cause
slowing down the search?

46. Consumer Group Coordinator
- Group coordinator is the broker which receives heartbeats(or polling
for message) from all consumers within the consumer group.
- Every consumer group has only one group coordinator.
- When a consumer want to join a consumer group, it sends a request to
group coordinator.

47. Consumer Group Leader
- The consumer group leader is one of the consumers in a consumer group.
- When a consumer wants to join a consumer group, it sends a JoinGroup
request to the group coordinator.
- The #irst consumer to join the group becomes the group leader.

48. Group Leader and Coordinator relationship
Rebalancing Event occurs when:
- A new consumer joins the consumer group.
- A consumer leaves the consumer group.
- A consumer is "disconnected" to consumer group (group coordinator
doesn't receive heartbeat event from this consumer)

49. Group Leader and Coordinator relationship
Rebalancing Event occurs when:
- A new consumer joins the consumer group.
- A consumer leaves the consumer group.
- A consumer is "disconnected" to consumer group (group coordinator
doesn't receive heartbeat event from this consumer)
- The group leader will make the assignment of each partition to each
consumer. Then the group leader sends this assignment to the group
coordinator.
- The group coordinator will send back the assignment to all consumers.
- Rebalancing event happens.

50. Group Leader and Coordinator relationship
Rebalancing Event occurs when:
- A new consumer joins the consumer group.
- A consumer leaves the consumer group.
- A consumer is "disconnected" to consumer group (group coordinator
doesn't receive heartbeat event from this consumer)
- The group leader will make the assignment of each partition to each
consumer. Then the group leader sends this assignment to the group
coordinator.
- The group coordinator will send back the assignment to all consumers.
- Rebalancing event happens.
Question: Why doesn't the group coordinator (broker) take the job that
assigns partitions to consumers, but the group leader (consumer) do that?

51. Group Leader and Coordinator
Algorithm:
1.Members JoinGroup with their respective subscriptions.
2.The leader collects member subscriptions from its JoinGroup response
and performs the group assignment.
3.All members (including the leader) send SyncGroup to #ind their
assignment.
4.Once created, there are two cases which can trigger reassignment:
a.Topic metadata changes that have no impact on subscriptions cause
re-sync. The leader computes the new assignment and sends
SyncGroup.
b.Membership or subscription changes cause rejoin.

52. Class Diagram

53. Delivery Semantic

54. Delivery Semantic
- At least once
- At most once
- Exactly once

55. Idempotent Producer

56. Idempotent Producer
- Generate PID for each producer
- PID and a sequence number are bundled together with the message

57. Isolation level for consumer
Read Committed
- non-transactional and COMMITTED transactional messages are visible.
- Read until LSO: Last Stable Offset
Read Uncommitted
- All messages are visible

58. Exactly Once Semantic
Part 1: Idempotent producer
- guarantee messages are produced once and in order.
Part 2: Producer Transaction support
- Commit or fail a set of produced messages and consumer offsets across
partitions
Part 3: Consumer transaction support
- Isolation level is read committed.
- Filter out messages for aborted transactions, and wait for transactions to
be committed before processing.

59. Exactly Once Semantic
Part 1: Idempotent producer
- guarantee messages are produced once and in order.
Part 2: Producer Transaction support
- Commit or fail a set of produced messages and consumer offsets across
partitions
Part 3: Consumer transaction support
- Isolation level is read committed.
- Filter out messages for aborted transactions, and wait for transactions to
be committed before processing.
Question: What if the producer is crashed, or the consumer is failed to
commit its processed offset.

60. Ka#ka Security

61. Authentication
TLS/SSL Integrating with PKI infrastructure
SASL/GSSAPI Integrating with Kerberos infrastructure
SASL/PLAIN Integrating with existing password server/database
Custom SASL
mechanism Integrating with existing authentication database
Others SASL/SCAM, SASL/OathBearer(beta), SASL_SSL

62. Authentication using SASL/GSSAPI with Kerberos

63. Authentication using SASL/PLAIN

64. Authorization using Zookeeper ACL

65. The Ecosystem

66. Ka#ka Connect
Ka#ka Connect is a framework for connecting Ka#ka with external
systems
e.g: databases, key-value stores, search indexes, and #ile systems.

67. Ka#ka Connect

68. Ka#ka Stream
Ka#ka Streams is a library for building streaming applications that
transform input Ka#ka topics into output Ka#ka topics.

69. Con#luent Schema Registry
- Con#luent Schema Registry provides a RESTful interface for storing
and retrieving Apache Avro® schemas.
- It stores a versioned history of all schemas.

70. Monitoring Tools
⁃ Ka#ka consumer lag checking: https://github.com/linkedin/Burrow
⁃ Ka#ka toolbox (Topic UI, Schema UI, Connect UI): https://www.landoop.com/lenses-box/
⁃ Ka#ka Manager: https://github.com/yahoo/ka#ka-manager
⁃ Zookeeper Con#iguration, Monitoring, Backup (Net#lix): https://github.com/soabase/exhibitor
⁃ Monitor the availability of Ka#ka clusters: https://github.com/linkedin/ka#ka-monitor
⁃ Ka#ka Tools: https://github.com/linkedin/ka#ka-tools
⁃ Cluster Management: https://github.com/apache/helix
⁃ Security Manager: https://github.com/simplesteph/ka#ka-security-manager
⁃ ....

71. Ka#ka vs RabbitMQ
Advanced Message Queuing Protocol
(RabbitMQ)
Log based Message Protocol
(Ka:ka)
A single
message is slow
to process
No problem
Holds up the processing of subsequent
messages in that partition.
Message Order
Cannot keep order in case of multiple
consumers
Can keep the order in the partition
Consumers
cannot keep up
with producers
when consumer die, we should carefully
delete any queues whose consumer has shut
down.
Otherwise, they continue consume memory,
affect other alive consumers.
Just store messages on queue (hard
drive)
We choose Ka:ka when:
- each message is fast to process and high message throughput
- message order is important
- Keyed messages
- Replay old messages
- Semantic Delivery

72. Client Implementation

73. Libraries
There are 2 main Ka:ka client libraries:
- https://github.com/Shopify/sarama
- https://github.com/con#luentinc/con#luent-ka#ka-go

74. Sarama
- Written in Golang. Open sourced by Shopify
- Easier to read. Missed many features

75. Con#luent Ka#ka Go
- Wrap again librdka#ka, which is written in C++.
- Have many latest features from Ka#ka.
- Still features behind of#icial Java Ka#ka client.
- Con#luent: founded by Ka#ka original authors.

76. Discussion about the Ka#ka PR
https://github.com/tsocial/ka#ka

77. References
Designing Data-Intensive Applications
Ka#ka The De#initive Guide
Ka#ka Streams in Actions

78. References
- http://ka#ka.apache.org/documentation.html
- Consumer Heartbeats Proposal
- https://cwiki.apache.org/con#luence/display/KAFKA/Offset+Management
- Exactly Once Delivery Proposal
- https://cwiki.apache.org/con#luence/display/KAFKA/Ka#ka+Replication
- https://cwiki.apache.org/con#luence/display/KAFKA/Ka#ka+Client-side+Assignment+Proposal
- https://github.com/edenhill/librdka#ka/issues/1308
- https://www.slideshare.net/jjkoshy/offset-management-in-ka#ka
- https://softwareengineeringdaily.com/2016/10/07/ka#ka-streams-with-jay-kreps/
- Logs: What every software engineer should know about real-time data unifying
- http://martin.kleppmann.com/2015/08/05/ka#ka-samza-unix-philosophy-distributed-data.html
- http://martin.kleppmann.com/2018/01/18/event-types-in-ka#ka-topic.html
- http://martin.kleppmann.com/2015/04/23/bottled-water-real-time-postgresql-ka#ka.html
- https://www.con#luent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-ka#ka/
- https://www.con#luent.io/blog/how-choose-number-topics-partitions-ka#ka-cluster
- https://engineering.linkedin.com/ka#ka/benchmarking-apache-ka#ka-2-million-writes-second-three-
cheap-machines

79. Q&A