Event Streaming in Retail with Apache Kafka

The Rise of Event Streaming in Retail
Use Cases, Architectures and Case Studies powered by Apache Kafka
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
linkedin.com/in/kaiwaehner

Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
The world is changing.

The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data

Stores become Software

Banks become Software

Taxis become Software
2
min

Cars become Software

Car Production becomes Software

This transformation is
happening everywhere

Retail

Software is Eating Retail
https://www.cbinsights.com/reports/CB-Insights_Retail-Tech-100-Webinar.pdf

Software is Eating Retail
14
https://www.cbinsights.com/reports/CB-Insights_Retail-Tech-100-Webinar.pdf

Real-time automation of customer interactions
Improved Shipping and
Delivery Methods
Customer-Driven
In-Store Experiences
Hybrid model
Shopping
Social Influencers / Virtual Reality Shopping:
Journey-focused innovation
General Trends:
● Highly competitive market, work to thin margins
● Moving from High Street (brick & mortar) to Online (Omni-Channel)
● Personalized Customer Experience - optimal buyer journey
Customer
Experience
(CX) Operational
Efficiencies
New
Business
Models
Disruptive
Trends
in
Retail
Warehouse logistics teams aligned
with real-time, in-store demands
Automating the supply chain
and core business processes
Data-Driven
Business Decisions and
Personalized Promotions

16
Increase Revenue
→ Customer Experience, Loyalty
Decrease Costs
→ Increase Operational Efficiency
Mitigate Risks
→ Regulatory Compliance
Reporting & Analytics
Stock inventory management
Legacy IT Replacement
(e.g. Middleware replacement)
Cyber Security
(incl. SIEM)
Fraud Prevention
(Anomalous credit card transactions)
Legacy IT Modernization
(e.g. Mainframe off-load / augmentation)
Example Retail Solutions for Event Streaming
Buyer journey - personalized offers
(Contextualized Next Best Action /
Targeted Offers)
Real-time Inventory Management
(across web, store, …)
Customer 360
(Call Center - Know Your Customer - KYC)
RT mobile applications
(clothing, food and beverage ordering, …)
Migration to the Cloud
(Hybrid on-prem / Cloud. Also Hybrid
Public Cloud vendors)
Microservices Architecture
Data Infrastructure layer
Business Application layer - the use cases
Data Pipelines Messaging
Microservice/
Event Sourcing Stream Processing Data Integration Streaming ETL Log Aggregation

What enables this
transformation?

Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

Real-time Data beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

Verticals are merging…
20
Source: Weekly FinTech Digest (Jan 2021) By Linas Beliūnas

This is a fundamental paradigm shift...
21
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

What is Event Streaming?

Event Streaming is the
Central Nervous System
for today’s enterprises.
Apache Kafka®
is the technology.

‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Banking
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka

Your Business as Streams of Events, powered by Kafka
Inventory
Shipping
Reporting
Orders
Frontend
Retail Example

The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

ETL/Data Integration Messaging
Highly Scalable
Durable
Persistent
Ordered
Real-time Difficult to Scale
No Persistence After
Consumption
No Replay
Batch
Expensive
Time Consuming

Highly Scalable
Persistent
Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Real-time
Highly Scalable
Durable
Persistent
Ordered
Real-time
Event Streaming

Kafka & Confluent

Confluent pioneered Event Streaming
Hall of Innovation
CTO Innovation
Award Winner
2019
Enterprise Technology
Innovation
AWARDS
Confluent founders are
original creators of Kafka
Confluent team wrote
80% of Kafka software
commits, has over 1M hours
technical experience with
Kafka, and operates 5,000+
clusters
Confluent Cloud
is the only multi-cloud, fully
managed, pay-as-you-go
event streaming service in
the world
Confluent Platform
completes Apache Kafka
and turns it into a secure,
enterprise-ready platform

Event Streaming with Kafka
in the Retail Industry?

Customer 360 and Real-Time Context-Driven Events
http://hyper-reality.co/
• CRM integration
• Real time product inventory
• Predictive recommendations
• Geo-location
• Payment integration
• …

Simplifying Omnichannel Retail at Scale
33
https://kafka-summit.org/sessions/simplifying-omni-channel-retail-scale/

“Walmart is a $500 billion in revenue
company, so every second is worth
millions of dollars. Having Confluent
as our partner has been invaluable.
Kafka and Confluent are the
backbone of our digital
omnichannel transformation
and success at Walmart.”
VP of Walmart Cloud

Real-Time Inventory System
https://www.confluent.io/blog/walmart-real-time-inventory-management-using-kafka/
https://www.confluent.io/kafka-summit-san-francisco-2019/when-kafka-meets-the-scaling-and-reliability-needs-of-worlds-largest-retailer-a-walmart-story/
● Investment in Kafka and Confluent has helped topline
company growth
● 8,500 nodes processing 11 billion events per day
● Deliver an omnichannel experience so every
customer can shop the way they want to

Innovative Business Modell
• Clothing rental subscription service
• Very different from a typical e-commerce model
• Need for a real-time event driven architecture
Benefits of serverless Confluent Cloud
• Cut launch time from over a year to 6 months
• Stable production ops set up in 1 week vs. 6 months
• Administrative overhead reduced by 10
36
https://www.confluent.io/customers/nuuly/

Realtime recommendations
Act at the right time with the right next best offer or action

Context-specific Customer 360
38
• Electrical retailer
• Hyper-personalized online retail
experience, turning each customer visit
into a one-on-one marketing
opportunity
• Correlation of historical customer
data with real-time digital signals
• Maximize customer satisfaction and
revenue growth, increased customer
conversions
https://www.confluent.io/customers/ao/

What is
Apache Kafka?

Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

Kafka Loves Your Existing Systems.
...many more
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect

Kafka Loves Your Existing Systems.
✓ Hundreds of ready-to-use
connectors available
✓ Supported by Confluent
or by partners (certified)
✓ Managed Connectors in
Confluent Cloud
$ confluent-hub install mongodb/kafka-connect-mongodb:1.3.0

Confluent Cloud
✓ Infinite Storage
Confluent Platform
✓ Tiered Storage (e.g., AWS S3)
Apache Kafka
• Tiered Storage upcoming
Kafka Stores Your Data Durably.
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1

Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

Kafka Streams — #1 popular choice
✓ Client library for Java, Scala
✓ Elastic, highly scalable, fault-tolerant
✓ Filter, transform, join, aggregate, window, etc.
✓ Exactly-once processing
✓ Full Kafka security integration
Build Apps, not Processing Clusters
✓ Standard JVM application
✓ Run in container, VMs, K8S, bare-metal, etc.
✓ No separate ‘processing cluster’ needed
✓ For S, M, L, XXL use cases
Kafka Makes Your Business Real-time.
Kafka Cluster
REAL-TIME APPLICATIONS

ksqlDB, the Event Streaming Database
DB
APP
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
TABLES & STATE
ksqlDB
DB One Solution for:
✓ capturing events
✓ stream processing
✓ serving both push
and pull queries
✓ also available in
Confluent Cloud
CREATE SOURCE CONNECTOR postgresSource WITH (...);
CREATE STREAM customer_changes …;
CREATE TABLE customers AS SELECT ... FROM customer_changes;
APP
“The easiest way to get started with Kafka.”

Event Topics Storage Partitions
Events / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
250,000 500 25,000 25
100 5 300 3
Kafka Scales with Your Business.

Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.

Distributed System with Replication and High Availability
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

Distributed System with Replication and High Availability
read,
write
App
(Kafka Streams)
Kafka
(data)
More Apps
(KSQL, Connect, Python,
REST, “You-name-it”)
Bookings
Team
Fraud
Team
…
Mobile
Team
…

Kafka
Connect
Kafka Cluster
Snowflake
Integration
Domain-Driven Design and Decoupled Microservices
SAP APO
Integration
Custom
Application
3rd Party
Middleware
Kafka-native Client
(Java, Python C++, KSQL, etc.)
Schema
Registry
Event Streaming Platform
Analytics Domain SCM Domain Customer Domain
Audit Logs,
RBAC, etc.

Architectures for
Apache Kafka in Retail?

Real-time Inventory System
C2
Time
P
C3
C1
C2
C3
Asset / People Tracking
Real-Time Alerting
Geofencing for
Distance Enforcement
Continuous
Real-Time Aggregation
Plan Future Location
Requirements
Batch Analytics
on Historical Events

Augmented reality with picture enhancement
Kafka for Large Messages
Pre-Processing
(Kafka Streams / ksqlDB)
Mobile App
Receive
enhanced
picture
Payment Service
including Fraud
Detection using
Machine Learning
(Kafka Streams)
Picture Enhancement
e.g. using image
processing software
(external service)
Send
picture

Global Central Nervous System

Customer Experience
(CX)
Profit -
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business Platform
Increase Operational
Efficiency
Migrate to Cloud
Mitigate Risk
(protect money)
Key Drivers Strategic Objectives
Risk & Cost Avoidance
Optimize Inventory
IoT - Device management
Optimize labor - Assets
Risk, What If & Analytics
Capacity Planning
State | Federal & Regulatory
Compliance | Governance - Data
Providence
Bottleneck Elimination
Audit, Regulatory
Digital
Transformation
Example Use Cases
$↑
$↓
$↔
Sales Demand Forecasting
$
Customer - Supply Chain Visibility
Event Streaming in the Retail Supply Chain

Mojix - Retail and Supply Chain IoT Platform
https://www.confluent.io/customers/mojix/
• Real-time operational
intelligence
• Across the edge and the
cloud
• Improves inventory accuracy,
enables operational
intelligence, supports
omnichannel sales

CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Customer
data
Train
schedule
Payment
data
Loyalty
information
Hybrid Retail Architecture

Point of Sale
(POS) Loyalty
System
Local Inventory
Management
Payment Discount
Customer
data
Train
schedule
Payment
data
Loyalty
information
Global Inventory
Management
Event Streaming at the Edge
in the Retail Store
Item Availability

Disconnected Edge
Time
P
C3 C2
C1
Context-specific
Advertisement
Real-time
(Milliseconds)
Location-based
Customer Action
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Payment Processing
Near Real-time
(Seconds)
Replication to Cloud
Batch
(Depending on Network Bandwidth)

Ship-Shore Highway – Swimming Retail Stores
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/

Event Streaming Is The Future Of Data
65
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Why Confluent?

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...

+13 services
combined into a single
platform in 6 months
3x customer
satisfaction increase vs.
prior contact us chat
300x traffic peaks
effortlessly handled
with zero downtime
Transforming our customer’s apps and data architecture
1 week vs +6 months
to have a mission-critical
ready environment
6 vs +12 months to
launch by focusing on
dev from day 1
0% Kafka
environment downtime
sitting at the core of the
business
17% increase in R&D
capacity with 1 FTE
freed to focus on
development vs. infra
1 year reduction in
time to market of large
real-time products
2x team capacity
increase by leveraging
ksqlDB side-by-side
Kafka Streams

Why Confluent?
Fully Managed Cloud Service
Self Managed Software FREEDOM OF CHOICE
COMMITTER-DRIVEN EXPERTISE Partners
Training
Professional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema
Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete Engagement Model
Revenue / Cost / Risk Impact
TCO / ROI
Executive Buyer

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Event Streaming in Retail with Apache Kafka

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Event Streaming in Retail with Apache Kafka

Similar to Event Streaming in Retail with Apache Kafka (20)

More from Kai Wähner

More from Kai Wähner (17)

Recently uploaded

Recently uploaded (20)

Event Streaming in Retail with Apache Kafka