FaaS composition using Kafka and Cloud-Events
LOCATION: Burton & Redgrave, DATE: November 7, 2019, TIME: 2:30 pm - 3:15 pm
https://serverlesscomputing.london/sessions/faas-composition-using-kafka-and-cloud-events/
Serverless functions or FaaS are all the rage. By leveraging well established event-driven microservice design principles and applying them to serverless functions we can build a homogenous ecosystem to run FaaS applications.
Kafka’s natural ability to store and replay events means serverless functions can not only be replayed, but they can also be used to choreograph call chains or driven using orchestration. Kafka also means we can democratize and organize FaaS environments in a way that scales across the enterprise.
Underpinning this mantra is the use of Cloud Events by the CNCF serverless working group (of which Confluent is an active member).
Objective of the talk
You will leave the talk with an understanding of what the future of cloud holds, a methodology for embracing serverless functions and how they become part of your journey to a cloud-native, event-driven architecture.
3. Origins of FaaS
●
●
●
●
“It’s easy to run without managing infrastructure. It will take care of
managing, scaling, monitoring and logging for you. Your applications run
milliseconds after the event has been triggered and thousands of
functions can work in parallel. You run the code only when needed.”
18. 1818
Rate of adoption
● Serverless is the top-growing extended cloud service for the
second year in a row, with a 50 percent growth over 2018
24 to 36 percent adoption.
● Stream processing is tied for first, increasing from
20 to 30 percent adoption.
Rightscale cloud report 2019
19. 1919
What is stream processing?
Event stream processing, or ESP, is a set of technologies designed to assist the
construction of event-driven information systems. ESP technologies include event
visualization, event databases, event-driven middleware, and event processing
languages, or complex event processing (CEP).
SELECT DataStream
Orders.TimeStamp, Orders.orderId,
Orders.ticker,Orders.amount, Trade.amount
FROM Orders
JOIN Trades OVER (RANGE INTERVAL '1' SECOND
FOLLOWING)
ON Orders.orderId = Trades.orderId;
https://en.wikipedia.org/wiki/Event_stream_processing
20. 20
Streaming processing as SQL
Stream
processor
STREAM
/user-reg
FILTER
SELECT users > 18
PROJECT
SELECT user.name
JOIN
SELECT u.name, a.country
from user-reg u JOIN
address a WHERE u.id = a.id
GROUP BY (TABLE)
SELECT u.country,
count(u.name) FROM user-reg u
GROUP BY u.country
WINDOW (TABLE)
SELECT u.country, count(u.name)
FROM user-reg u WINDOW TUMBLING
(SIZE 1 MIN) group by u.country
STREAM
/address
21. 21
1. Stream processor runs ‘future’ on
local-state of item-bids
auction items
3. Rejoined
removes item
4. Stream triggers FaaS
complete item processor
bid-history
Stream table join
5. FaaS Notify all bidders
bid-notifications
2.
Rewrite ‘completed’ item
status to retrigger join
item status
24. 24
{
StreamsBuilder builder = new StreamsBuilder();
KStream<String, Payment> incoming = builder.stream(TASK_TOPIC);
incoming.flatMapValues(payment -> {
ArrayList<String> fanTasks = new ArrayList<>();
for (int i = 0; i < payment.getAmount(); i++) {
fanTasks.add("fan-" + i);
}
return fanTasks;
}
).to(TASK_FANOUT); flatMapValues: Many V to the same K
Or
flatMap: Many K,V to one K,V pair
28. 2828
A Stream processing DataFlow is
both workflow and choreography
(EDA, SEDA, Event-driven, streaming)
29. 2929
Choreography
1. Fire and forget / Async events
2. Producers emit state (not commands)
3. Consumers react to events
4. There is no master plan, no central brain
LOOSE COUPLING
34. 34
Evolvability
user experience?
how many users
affected?
has it happened
before?
new old
supports data change, logic change, logic extension, schema
evolution, loose coupling, add processors, A/B path
35. 35
is a distributed event streaming platform
Publish & Subscribe
to Events
Store
Events
Process & Analyze
Events
52. 52
● CNCF
● Version 1.0 October 2019
● Specification (an Event envelope)
● Transport Bindings (Kafka, AMQP, HTTP others)
● SDKs: Java, .NET, Go-lang, Javascript etc
● Useful for exposing to external apps (Events as APIs)
CloudEvents is a specification for describing event data in common formats
to provide interoperability across services, platforms and systems.
53. 5353
CloudEvent semantics
● Self describing / self contained / stand alone
● Suppressed / non-intruisive
● Language, transport, data format independent
● Propagation
● Extensions
● Batteries included