Apache Kafka Meetup Japan #2 https://kafka-apache-jp.connpass.com/event/45923/ 発表資料

1. Queryable State for Kafka Streams
/ @laclefyoshi / ysaeki@r.recruit.co.jp

2. •
• Kafka Streams
• State for Kafka Streams
• Queryable State for Kafka Streams
•
•
2

3. • 2011/04
• 2015/09
•
• Druid (KDP, 2015)
• RDB NoSQL ( , 2016; : HBase )
• ESP8266 Wi-Fi IoT (KDP, 2016)
•
• (DEIM 2014)
• (WebDB Forum 2014)
• Spark Streaming (Spark Meetup December 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• (FutureOfData; 2016)
3

4. Apache Storm
PipelineDB
Druid
Apache Kafka
4

5. Apache Storm
Memcached
Redis
Apache Kafka
5

6. Apache Kafka
6

7. Kafka
7

8. Apache Kafka Version 0.10.1.0
• New Features
• [KAFKA-1464] - Add a throttling option to the Kafka replication tool
• [KAFKA-3176] - Allow console consumer to consume from
particular partitions when new consumer is used.
• [KAFKA-3492] - support quota based on authenticated user name
• [KAFKA-3776] - Unify store and downstream caching in streams
• [KAFKA-3858] - Add functions to print stream topologies
• [KAFKA-3909] - Queryable state for Kafka Streams
• [KAFKA-4015] - Change cleanup.policy config to accept a list of
valid policies
• [KAFKA-4093] - Cluster id
8

9. Kafka Streams
• Version 0.10.0.0
• Kafka API
9

10. Kafka Streams
Stateless transformations Stateful transformations
10

11. State for Kafka Streams
• Stateful transformations
•
11

12. State
• Kafka Streams
• 2
• State Store RocksDB
• Kafka StateChangelog
• Kafka Streams
• Key Value
• Window
12

13. • Record Stream
• Changelog Stream
• State (Table)
Record Changelog State
13

14. Queryable State for Kafka Streams
• Version 0.10.1.0
• State Store API
14

15. Queryable State
• ReadOnlyKeyValueStore
• Window Streams State
• Key
• Value
• ReadOnlyWindowStore
• Window Streams State
• Key, From (Long), To (Long)
• Iterator<KeyValue<Long, Value>>
15

17. Kafka Streams
• Stateful transformations count
KafkaStreams streams = null;
KStreamBuilder builder = new KStreamBuilder();
KStream<String, String> textLines =
builder.stream(stringSerde, stringSerde, inputStream);
KStream<String, Long> wordCounts =
textLines
.flatMapValues(value -> ...)
.groupBy((key, word) -> word)
.count(storeName)
.toStream();
wordCounts.to(stringSerde, longSerde, outputStream);
streams = new KafkaStreams(builder, conf);
17

18. Queryable State Streams
• Streams o.a.k.streams.KafkaStreams
KafkaStreams streams = ...;
String key = "kafka";
ReadOnlyKeyValueStore<String, Long> store =
streams.store(
storeName,
QueryableStoreTypes.<String, Long>keyValueStore());
Long value = store.get(key);
return value;
18

19. Queryable State Streams
• API
KafkaStreams streams = ...;
get("/results/:key",
(req, res) -> {
String key = req.params(":key");
ReadOnlyKeyValueStore<String, Long> store =
streams.store(
storeName,
QueryableStoreTypes.<String, Long>keyValueStore());
Long value = store.get(key);
return value;
});
19

20. github:laclefyoshi/kafka_streams_example

21. Apache Kafka
21
HTTP

22. $ java -cp target/kafka_streams_example-1.0-SNAPSHOT.jar
org.saekiyoshiyasu.App
kafka -> null
...
kafka -> 5
$ ./bin/kafka-console-producer.sh --topic input-stream
--broker-list 127.0.0.1:9092
What is Kafka good for?
...
22

23. Queryable State for Kafka Streams
•
Kafka Streams ?
23

24. Queryable State for Kafka Streams
•
Kafka Streams ?
State
24

25. • Queryable State
• Kafka Streams
25