Apache Kafka Meetup Japan #5 @LINE での発表資料です。

1. Custom management apps for Kafka
2018/12/18
「Apache Kafka Meetup Japan #5 @LINE」
Kimura, Sotaro（@kimutansk）
https://www.flickr.com/photos/wwworks/8693876413/

2. I am...
• Kimura, Sotaro(@kimutansk)
– Data Engineer IT choreman at Dwango
• Data Analytics Infrastracture team
• On premiss～Cloud, SRE～Individual function
• Batch / Stream processing, developer / leader
– In other words, IT choreman
– Language, Scala/Java/Python/Ruby/Go etc...
– Favorite
• Stream data processing
• Kafka, Flink, Fluend, Spark, Pulsar etc...

3. Systems of our team abstract
Otherteam’s system
Explanatorynotes
Externalreq/res
Internalreq/res
Job execution
Monitor/Manage

4. Systems of our team abstract
Otherteam’s system
Explanatorynotes
Externalreq/res
Internalreq/res
Job execution
Monitor/Manage

5. Kafka monitoring/manage funcs
• Topic setting manage
• User account / ACLs manage
• Topic list documents page generate
• Resource monitoring
– CPU utilization, Memory usage, Disk usage, IO wait
– Network processor / Request handler idle rates
• Flow manage
– Auto recommend expanding partitions

6. Kafka monitoring/manage funcs
• Topic setting manage
• User account / ACLs manage
• Topic list documents page generate
• Resource monitoring
– CPU utilization, Memory usage, Disk usage, IO wait
– Network processor / Request handler idle rates
• Flow manage
– Auto recommend expanding partitions
Today’s target.

7. Topic setting manage
• Operating kafka from CLI
– Topic create
– Topic partition expand
– Topic specific setting
$ ./kafka-topics.sh --zookeeper kafka01:2181--create --topic sample_topic01
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --partitions 3
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --config cleanup.policy=compact

8. Topic setting manage
• Operating kafka from CLI
– Topic create
– Topic partition expand
– Topic specific setting
$ ./kafka-topics.sh --zookeeper kafka01:2181--create --topic sample_topic01
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --partitions 3
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --config cleanup.policy=compact
Too hard of human’s operation !

9. User account / ACLs manage
• Premise
– We decided to use SSL for Kafka AuthN/ACLs manage.
• Because, we investigated it at about 2017/04.
• At that time, ruby-kafka didn’t support SASL.
– In ruby-kafka v0.4.0, supported ：）
– In our team, AuthN is not applied.
• AuthN is only applied to the access from other teams.

10. User account / ACLs manage
• Operating kafka from CLI
– Add consumer permission to user
– Add producer permission to user
– Remove producer permission from user
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-consumer01" --consumer --
topic sample_topic01 --group test_group01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-producer01"--producer --
topic sample_topic01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--remove --allow-principal"User:CN=test-producer01" --producer
--topic sample_topic01

11. User account / ACLs manage
• Operating kafka from CLI
– Add consumer permission to user
– Add producer permission to user
– Remove producer permission from user
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-consumer01" --consumer --
topic sample_topic01 --group test_group01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-producer01"--producer --
topic sample_topic01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--remove --allow-principal"User:CN=test-producer01" --producer
--topic sample_topic01
These commands confuse us!

12. Our needs
• Operators don’t execute raw command.
• Topic/ACLs settings are managed by file.
– To enable reviewing each other.
• So...

13. The administrative client for Kafka
• org.apache.kafka.clients.admin.AdminClient
– Through Broker API Endpoint
– Introduced in 0.11.0.0
– Manage below elements.
• Broker Configuration
• Topic
• Partition
• Topic Configuration
• ACLs
• ReplicaLogDir

14. Developed apps(Topic manage)
• KafkaTopicManageApp
– Using org.apache.kafka.clients.admin.AdminClient
– Process flow
• Load application settings, topic settings.
• Get topic information from kafka cluster.
• Create topic with specified settings.
• Update topic with specified settings.
– Applying settings is idempotent, so app updates always.

15. Developed apps(Topic manage)
• KafkaTopicManageApp
– Application setting
bootstrapServers:
- "kafka01:9092"
- "kafka02:9092"
- "kafka03:9092"
defaultPartitions: 1
defaultReplicationFactor: 3

16. Developed apps(Topic manage)
• KafkaTopicManageApp
– Topic setting
topics:
- name: TestTopicName
partitionNum: 4
config:
min.insync.replicas: 2
segment.bytes: 104857600

17. Developed apps(ACLs manage)
• KafkaAclManageApp
– Using org.apache.kafka.clients.admin.AdminClient
– Process flow
• Load application settings, acl settings.
• Apply acls to kafka cluster.
– Applying settings is idempotent, so app applys always.

18. Developed apps(Topic manage)
• KafkaAclManageApp
– Acl setting
userAcls:
- name: "CN=test_user01"
authorized_consumer_topics:
- "sample_topic01"
- "sample_topic02"
specific_consumer_group:
- "test_user01_group01"
- "test_user01_group02"

19. Apps operation
• Settings are managed by Github.
• Update flow
– Update settings.
– PR & review
– Execute apps from jenkins.
• Check flow
– Check difference between settings and kafka cluster
periodically.
– If differences exist, notify to operators.(Slack)

20. Conclusion
• Kafka cluster operation commands confuse us.
• So, let’s automate by kafka manage tools or
your hand made apps with AdminClient!

21. Thank you for your attention!
https://www.flickr.com/photos/savannahcorps/7409364642