Slides from my talk "If an event is published to a topic and no one is around to consume it, does it make a sound?" at Kafka Summit Live in August 2020.
See recording here: https://www.confluent.de/resources/kafka-summit-2020/if-an-event-is-published-to-a-topic-and-no-one-is-around-to-consume-it-does-it-make-a-sound/
Kafka Summit 2020: If an event is published to a topic and no one is around to consume it, does it make a sound?
1. If an event is published to a topic
and no one is around to consume it,
does it make a sound?
Bernd Ruecker
Co-Founder and Chief Technologist of Camunda
@berndruecker
4. SELECT name
FROM people
WHERE city = `Berlin´
reader = new BufferedReader(new FileReader("people.txt"));
while ((row = reader.readLine()) != null) {
String[] data = row.split(",");
if ("Berlin".equals(data[3])) { // city
System.out.println(data[1]); // name
}
}
reader.close();
@berndruecker
7. Functional programming, reactive & streaming = the greatest thing since sliced bread!
… the term Event Stream becomes widely
adopted and promotes a new mental model
of how the systems are implemented.
… the main abstraction in use is a
composition of [stateless] functions
performing transformation on immutable
events. The output of an overall system can
be modeled as the application of those
composed functions on the input stream.
https://medium.com/holisticon-consultants/from-stream-to-state-eb1cade5d403
@berndruecker
9. Connor Riley
#Cisco just offered me a job!
Connor Riley
#Cisco just offered me a job! Now I
have to weigh the utility of a fatty
paycheck against the daily commute to
San Jose and hating the work.
@berndruecker
10. #Cisco just offered me a job!
Connor Riley
#Cisco just offered me a job! Now I
have to weigh the utility of a fatty
paycheck against the daily commute to
San Jose and hating the work.
Event goes into the stream
Stateless Stream Processor
Filter all tweets
with #cisco
Stateless Stream Processor
Find offensive
tweets
Tim Levad (Cisco employee)
Who is the hiring manager? I’m sure
they would love to know that you will
hate the work. We here at Cisco are
versed in the Web.
Action
Input
Output
→ Job offer revoked
@berndruecker
13. #Cisco just offered me a job!
Connor Riley
#Cisco just offered me a job! Now I
have to weigh the utility of a fatty
paycheck against the daily commute to
San Jose and hating the work.
Event goes into the stream
Stateless Stream Processor
Filter all tweets
with #cisco
Refine / change logic
Find offensive
tweets
Tim Levad (Cisco employee)
Who is the hiring manager? I’m sure
they would love to know that you will
hate the work. We here at Cisco are
versed in the Web.
Action
Input
Output
@berndruecker
14. #Cisco just offered me a job!
Connor Riley
#Cisco just offered me a job! Now I
have to weigh the utility of a fatty
paycheck against the daily commute to
San Jose and hating the work.
Event goes into the stream
Stateless Stream Processor
Filter all tweets
with #cisco
Find offensive
tweets
Tim Levad (Cisco employee)
Who is the hiring manager? I’m sure
they would love to know that you will
hate the work. We here at Cisco are
versed in the Web.
Action
Input
Output
@berndruecker
16. #Cisco just offered me a job!
Stateless Stream Processor
Filter all tweets
with #cisco
tweets
Filter tweets with
similar content
within 15 minutes
time
Stateful Stream Processor
State is
in the
API!
@berndruecker
17. #Cisco just offered me a job!
Stateless Stream Processor
Filter all tweets
with #cisco
tweets
Filter tweets from
our top community
members
Stateful Component
(as stateless stream processor)
State is in the
component
@berndruecker
18. #Cisco just offered me a job!
tweets
Enrich tweet with
personal information
of sender
lookup
Bad, because
- Load on API
- Availability
- Other disadvantages of synchronous remote communication
@berndruecker
19. #Cisco just offered me a job!
tweets
Person stream
Stateful Stream Processor
Enrich tweet
@berndruecker
20. #Cisco just offered me a job!
tweets
Stateful Stream Processor
Persons that tweeted
Enrich tweet
Gather personal
info
@berndruecker
21. #Cisco just offered me a job!
tweets
Stateful Stream Processor
THIS is
workflow logic!
Workflow
engines can
hold state
Enrich tweet
Gather personal
info
@berndruecker
Persons that tweeted
22. #Cisco just offered me a job!
tweets
Stateful Stream Processor
Workflow
engines can
hold state
Enrich tweet
Gather personal
info
Publish /
subscribe
Mitigates problems of synchronous communication!
@berndruecker
23. Example
workflowClient.newWorker()
.jobType("person-lookup")
.handler((jobClient, job) -> {
System.out.println("Gather personal information");
jobClient.newCompleteCommand(job.getKey()).send().join();
}).open();
publish / subscribe
@berndruecker
client.newCreateInstanceCommand()
.bpmnProcessId("enrich-tweet")
.variables(...)
.latestVersion()
.send().join();
client.newCreateInstanceCommand() .bpmnProcessId("order-process") .latestVersion() .send() .join();
Or use the Zeebe Kafka Connector
26. Where do we stand now?
State goes into
Stream
Processor API
Component
Workflow Engine*
*helps to implement common patterns in stateful components
@berndruecker
27. Where do we stand?
State goes into
Stream
Processor API
Component
Workflow Engine
This translation
requires additional
thought!
@berndruecker
42. Summary
• Streams need to lead to actions outside the control of the
streams architecture (otherwise nothing has really happened)
• Where to handle state is a difficult question
• Streams vs. Workflow might be about declarative vs.
imperative thinking (what does your team understand?)
• But streams and workflows also go hand in hand. Getting
from stream to workflow involves a translation component
@berndruecker