Message Driven and Event Sourcing

Message Driven & Event Sourced
Andrea Gioia - Partner Quantyca
Paolo Castagna - Account Executive, Confluent
Message Driven & Event Sourced / quantyca.it | confluent.io

Scalability & Cloud / quantyca.it | google
Andrea Gioia
Partner, Quantyca
Paolo Castagna
Account Executive, Confluent

Intro

Message driven
Strong decoupling
La necessità di delineare per ogni componente il giusto confine che possa garantire il più basso accoppiamento con gli
altri, l’isolamento e la trasparenza sul dislocamento richiede di basare le interazioni all’interno delle moderne data
platform sullo scambio asincrono di messaggi.
Broker

Inside the broker
Smart endpoints and dumb pipes
Attenzione a non spostare la complessità all’interno del broker. Le logiche di business dovrebbero essere il più possibili
contenute nelle componenti che accedono al broker non nel broker stesso.
=
Big ball of mud Spaghetti in a box

Due tipologie di broker
Service Bus e Data Bus

Service Bus (orchestration over choreography)
Nelle tradizionali architettura SOA il coordinamento dei servizi esposti
da ogni sistema avviene tramite un Enterprise Service Bus (ESB)
centralizzato.
L’architettura basata su ESB permette di:
1. comporre servizi esposti da sistemi legacy
2. gestire transazioni distribuite
3. creare un anticorruption layer per isolare i sistemi monolitici
4. Facilitare il test e monitoraggio dell’infrastruttura

Data Bus (choreography over orchestration)
Nelle moderne architettura a microservizi si rinuncia alla
centralizzazione per aumentare la scalabilità del sistema.
Ogni servizio è modellato intorna ad uno specifico dominio di business
ed è completamente autonoma dagli altri dal punto di vista di :
1. Implementazione
2. Storage
3. Deploy
L’allineamento a livello di dati tra i vari servizi avviene tramite un data
bus basato su una piattaforma di event streaming distribuita (Kafka)

Architettura Ibrida e API Manager
Le due architetture possono coesistere. Il layer di servizi avrà sempre
più una doppia polarità:
● da una parte servizi esposti da sistemi legacy orchestrati tramite
ESB (SOA)
● dall’altra servizi con coordinamento decentralizzato e altamente
scalabili (Microservices)
Per i consumatori di servizi questa architettura ibrida non sarà
direttamente visibile in quanto l’accesso ai servizi passerà tramite API
Gateway che medieranno l’accesso ai servizi reali

How Apache Kafka and Confluent
enable event streaming
architectures

Events
An immutable record that something as some point has happened
A Sale An Invoice A Trade A Customer
Experience

5.2 Million Citizens
Norwegian Work and
Welfare Administration
Life is a Stream of Events

Pre-Streaming

Request-Response Applications
App
Service
Service
Service Service
Service
Service
Service
Service
App
■ DETERMINISTIC
■ RIGID
■ TIGHT

Event-Driven Applications
App
App
Developer
APIs
Service
Service
Service
Service
Service
Streaming
Platform
■ RESPONSIVE
■ FLEXIBLE
■ EXTENSIBLE

Moving from Pre-Streaming to Event Driven Architectures
Request-Response vs Event-Driven: You Need Both!
Request-Response Event-Driven

Event Centric Thinking
All Your Data is a Stream of Events

The Event Streaming Platform

Why Didn’t it Work Before?
Past solutions turned out to be insufficient
Message-Oriented Middleware
No persistence
Single point of failure
Not fault tolerant
Cannot order messages
Cannot process messages in flight
Order of magnitude lower throughput
No “replay” functionality
EAI & ESBs
Not event-oriented
Fragile and bespoke
Weak transformation
capabilities
ETL
Slow and batch oriented
Point-to-point, not
publish & subscribe
Not a true infrastructure
platform

The World Has Changed
Apache Kafka is at the core of these industry trends and, ultimately,
the digitalisation of every business
Microservices Mobile Machine
Learning
Internet of
Things

The Event Streaming Platform
We saw that picture before, what’s different now?
■ APACHE KAFKA CAN STORE EVENTS
Kafka can store events for months, years, or indefinitely if useful. The limit is the amount of physical
storage available on Kafka brokers. Events are stored in order on a per partition basis and they can be
replayed and reconsumed (in order) whenever necessary.
■ EVENTS ARE STORED IN ORDER (PER PARTITION)
Order of events is preserved and the ability to operate on a ‘stream’ level, also via Kafka Streams APIs
or KSQL, provides a powerful way to simplify data integration, data cleansing, data transformations
tasks as well as use Kafka to exchange and publish entire datasets. With Kafka, events are first class
citizens, but also the full history and state is available to applications (it’s like having a database and a
messaging system at the same time).
■ KAFKA CAN EASILY OPERATE A COMPANY SCALE
Scalability, fault-tolerance, elasticity, and multi-tenancy are built-in capability.

Data Streaming and Maturity Stages
Value
Maturity (Investment & time)
2
Enterprise
Streaming Pilot /
Early Production
LOB(s) Pilot(s); Small
teams experimenting;
pub/sub / integration.
→ 1-3 use cases quickly
moved into Production -
but fragmented.
Pub + Sub Store Process
5
All data in the organization
managed through a single
Streaming Platform.
→ Digital natives / digital
pure players - probably using
Machine Learning & AI
(Relational databases -
redundant).
Central Nervous
System
1
Developer
Interest
Developer downloads
Kafka & experiments
(15 mins on laptop).
Pre-Streaming
Legacy systems.
Batch processes;
→ Complex
→ Slow / Silo’d
→ Expensive
4
Global
Streaming
Streaming Platform
managing majority of
mission critical data
processes, globally, with
multi-datacenter replication
across on-prem and hybrid
clouds. In parallel with
other Big Data
infrastructure.
3
SLA Ready,
Integrated
Streaming
Multiple mission critical
use cases in production,
with; scale, DR & SLAs.
→ Streaming clearly
delivering business
value, with C-suite
visibility.
Projects
Platform

Designing Event-Driven Systems
In this book Ben explains how service-based architectures and stream processing tools
such as Apache Kafka can help you build business-critical systems.
● Understand why replayable logs such as Kafka provide a backbone for both service
communication and shared datasets
● Explore how event collaboration and event sourcing patterns increase safety and recoverability
with functional, event-driven approaches
● Apply patterns including Event Sourcing and CQRS, and how to build multi-team systems with
microservices and SOA using patterns such as “inside out databases” and “event streams as a
source of truth”
● Build service ecosystems that blend event-driven and request-driven interfaces using a
replayable log and Kafka's Streams API
● Scale beyond individual teams into larger, department- and company-sized architectures, using
event streams as a source of truth
The book is available for free in PDF from the Confluent website!
https://www.confluent.io/designing-event-driven-systems

Royal Bank of Canada (RBC)
We saw that picture before, what’s different now?
https://www.confluent.io/customers/rbc
■ 16 Million Clients
■ 35 Countries
■ 30+ Use-cases
■ 50+ apps
■ 10+ lines of businesses
“Kafka transforms even the most basic of initiatives. Adoption of Kafka at
RBC has been massive and organic. Within the first six weeks after our
launch of Kafka, we had 37 teams asking to use Kafka for various
projects and initiatives.”
-- Kerry Joel, Senior Director,
Product Innovation, Data and Analytics
Digital
Marketi
ng
Securit
y
Consumer
Credit
Services
SaaS
Corporate
Real Estate
Investor
Services
Treasury
Services
….
FraudData
Wareho
use
Microservices
https://www.youtube.com/watch?v=WTxmHHJcHRc

The Streaming Platform
Technical Capabilities
Publish &
Subscribe
Store Process

Takeaways

In conclusione
Elementi chiave
1. Service integration: esb, microservizi, api manager
2. Service portfolio
3. Competence portfolio
4. Piano di migrazione evolutivo
Prossimi passi
1. Scalabilità
2. Governance non solo a livello dati ma anche a livello servizi

Corso Milano, 45 / 20900 Monza (MB)
T. +39 039 9000 210 / F. +39 039 9000 211 / @ info@quantyca.it
www.quantyca.it

Message Driven and Event Sourcing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Message Driven and Event Sourcing

Similar to Message Driven and Event Sourcing (20)

More from Paolo Castagna

More from Paolo Castagna (9)

Recently uploaded

Recently uploaded (20)

Message Driven and Event Sourcing