Supply Chain Optimization with Apache Kafka

Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de
Apache Kafka for Optimization of
Supply Chain Management (SCM)
Decoupled Microservices, Data Integration, Real-Time Analytics, and More…
Kai Waehner
Field CTO
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

Apache Kafka for Supply Chain Management (SCM) - @KaiWaehner - www.kai-waehner.de
Supply Chain Management (SCM)
• Planning and coordination of all the people,
processes, and technology involved in creating
value for a company
• Cross cutting processes, including purchasing /
procurement, logistics, operations / manufacturing,
and others
• Automation, robustness, flexibility, hybrid
deployment (edge + cloud)
Six Sigma (6σ)

SCOR (Supply-Chain Operations Reference Model)

The Future of SCM – Outlook to 2040
https://www.ipa.fraunhofer.de/en/press-media/press_releases/how-supply-chain-management-will-change-by-2040.html
Autonomy: Vehicles, machines, sensors will be
handled largely autonomously
Connected: SCM will no longer form a chain, but
rather a network:
Communication: The number of people involved will
be rising because of the increasing number of new
goods and services being exchanged.
Fast: Companies will develop into significant
players within a shorter space of time, although by
the same token they can also disappear all the
faster from the market.
Flexible: SCM processes will be able to adjust more
quickly to unforeseen events such as pandemics,
wars and natural disasters.

Time Frames are Shorter
Source:

Rapid Change
Source:

Historical Models are no Longer Viable
Source:

Chronic Lack Of Visibility
Source:

Huge demand to build an open, flexible, scalable platform
• Real time
• Scalability
• High availability
• Decoupling
• Cost reduction
• Flexibility
• Elasticity
• Standards-based
• Extendibility
• Security
• Infrastructure-independent
• Multi-region / global

Event Streaming
with
Apache Kafka

STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming

An Event Streaming Platform
is the Underpinning of an Event-driven Architecture
MES
ERP
Sensors
Mobile
Customer 360
Real-time Alerting
System
Data warehouse
Producers
Consumers
Streams of real time events
Stream processing apps
Connectors
Connectors
Stream processing apps
Supplier
Alert
ForecastInventory Customer
Order

Hybrid and Multi-Cloud Architectures
VM
SELF MANAGED
FULLY MANAGED
Edge vs. regional vs. global deployments
Cloud-first (greenfield) vs.
Hybrid architecture vs.
Strategic move from on premise to cloud

Why Apache Kafka for Supply Chaim Management (SCM)?
• Real-time messaging (at scale, mission-critical)
• Global Kafka (edge, data center, multi-cloud)
• Cloud-native (open, flexible, elastic)
• Data integration (legacy + modern protocols, applications,
communication paradigms)
• Data correlation (real-time + historical data, omni-channel)
• Real decoupling (not just messaging, but also infinite
storage + replayability of events)
• Real-time monitoring
• Transactional data and analytics data (MES, ERP, CRM,
SCM, …)
• Applied machine learning (model training and scoring)
• Cybersecurity
• Complementary to legacy and cutting-edge technology
(Mainframe, PLCs, 3D printing, augmented reality,
…)

Customer Experience
(CX)
Profit -
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business Platform
Increase Operational
Efficiency
Migrate to Cloud
Mitigate Risk
(protect money)
Key Drivers Strategic Objectives
Risk & Cost
Avoidance
Optimize Inventory
IoT - Device management
Optimize labor - Assets
Risk, What If & Analytics
Capacity Planning
State | Federal & Regulatory
Compliance | Governance - Data
Providence
Bottleneck Elimination
Audit, Regulatory
Digital
Transformation
Example Use Cases
$↑
$↓
$↔
Sales Demand Forecasting
$
Customer - Supply Chain Visibility
Use Cases for Event Streaming in the Supply Chain

Food Value Chain
IoT-Based and Data-Driven
Single source of truth
across the food value chain
(in the factories, and across regions)
Business critical
operations
(tracking, calculations, alerts, …)
https://www.confluent.io/blog/creating-iot-based-data-driven-food-value-chain-with-confluent-cloud/

Architectures and Use Cases
Decoupled
Microservices

Kafka Connect
Kafka Cluster
MQTT
Integration
Domain-Driven Design (DDD) for your Event Streaming Platform
SAP
Integration
Real Time
Predictions
IoT Platform
Connector
Java / Python /
”you-name-it”
Schema
Registry
Event Streaming Platform
IoT Domain OT Domain Analytics Domain
 Independent and loosely coupled, but scalable, highly available and reliable!
21

Decoupled Microservices @ Porsche
https://medium.com/porschedev/apache-kafka-at-porsche-literary-figure-meets-car-manufacturer-ead9d99c3bc
https://medium.com/next-level-german-engineering/data-streaming-porsche-bc49c6aa17a8
“The recent rise of data streaming has opened new possibilities for real-time analytics. At Porsche, data
streaming technologies are increasingly applied across a range of contexts, including warranty and sales,
manufacturing and supply chain, connected vehicles, and charging stations.”
Sridhar Mamella (Platform Manager Data Streaming at Porsche)

Standard Software +
Custom Applications

SCM
=
Zoo of
technologies and
products
Options:
No software
Buy
Rent
Make
Mix
TMS
Transport Management System
WMS
Warehouse Management System
WES
Warehouse Execution System
DPS
Demand Planning System
DRP
Distribution Requirements Planning
LMS
Labor Management System
CRM
Customer Relationship Management
SRM
Supplier Relationship Management
ERP
Enterprise
Resources
Planning
BI
Business Intelligence
MES
Manufacturing Execution System
PLM
Product Lifecycle Management

Postmodern SCM / ERP (coined by Gartner) built with Kafka + other Apps
Replace legacy, monolithic and highly customized ERP suites
by a mixture of loosely coupled, exchangeable cloud-based and on-premises applications.
TMS
Legacy Proprietary
SOAP Web Services
Supplier
Alert
ForecastInventory Customer
Order
Core ERP
CRM
SaaS
Kafka Interface
MES
Proprietary
HTTP Web Services
LMS
Legacy Homegrown
Database + CDC
SRM
Kafka-native

Postmodern SCM / ERP built with Kafka
Zero downtime with rolling upgrades and backwards compatibility
MirrorMaker 2 /
Confluent Replicator
Tier 1 Supplier
Server: Latest version
Clients: 0.11, 2.0
Streaming replication
between
SCM applications
Version compatibility between
different
clients and servers
Core ERP
Server: AK 2.3
Clients: 2.0, 2.5
MES
Server: CP 54 / AK 2.4
Clients: 0.11, 2.4
Confluent
Cluster Linking

Real-Time Inventory System
https://www.confluent.io/blog/walmart-real-time-inventory-management-using-kafka/
https://www.confluent.io/kafka-summit-san-francisco-2019/when-kafka-meets-the-scaling-and-reliability-needs-of-worlds-largest-retailer-a-walmart-story/
“Retail shopping experiences have evolved to include multiple channels, both online
and offline, and have added to a unique set of challenges in this digital era. Having
an up to date snapshot of inventory position on every item is a very important aspect
to deal with these challenges. We at Walmart have solved this at scale by designing
an event-streaming-based, real-time inventory system leveraging Apache Kafka…
Like any supply chain network, our infrastructure involved a plethora of event sources
with all different types of data”
Suman Pattnaik, Big Data Architect @
Walmart

Improving the
SCOR Metrics

Optimization of the Supply Chain Process

Supply Chain Planning
Streaming and Batch Analytics Use Case
• Prepare for the unknown and the
unknowable
• Act at the right time in the right context
• Yossi Sheffi: “You need to have
sensors in the ground”, then SCM
becomes a process of sensing and
responding
What if…?

From a batch-oriented systems architecture to
a streaming micro services platform
Thoughtworks:
https://www.confluent.io/kafka-summit-sf17/Fast-Data-in-Supply-Chain-Planning

Strangler Design Pattern
The Big Bang does not always work…
“The most important
reason to consider a
strangler fig
application over a cut-
over rewrite is
reduced risk.”
Martin FowlerThoughtworks:
https://www.confluent.io/kafka-summit-sf17/Fast-Data-in-Supply-Chain-Planning

Supply Chain Purchasing
Real-time Natural Language Processing (NLP) for Digital Contract Intelligence

Apache Kafka as Infrastructure for ML

Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams/
ksqlDB
Kafka Connect
Confluent REST Proxy
Confluent Schema Registry
Go/.NET/Python
Kafka Producer
ksqlDB
Python
Client

BMW Group
Industry-ready NLP Service Framework Based on Kafka
https://www.confluent.io/kafka-summit-lon19/industry-ready-nlp-service-framework-kafka/

Digital Twin +
Digital Thread

Digital Twin – Merging the Physical and the Digital World
38
• Downtime reduction
• Inventory management
• Fleet management
• What-if simulations
• Operational planning
• Servitization
• Product development
• Healthcare
• Customer experience
“Virtual representation of something else (Physical thing, process, service)”
“A living model that drives a business outcome”
https://www.youtube.com/watch?v=Ri0TD7kYsIQ

Software and Digital Services become the Key Differentiator
39
https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery https://www.rolls-royce.com/media/press-releases-archive/yr-2012/121030-the-hour.aspx

Digital Thread
40
Digital Twin vs. Digital Thread?
I only use the term Digital Twin in the following slides.
Both terms overlap, often meaning the same.
Span
the
entire
lifecycle

Apache Kafka
Kafka as Integration Platform
for the Digital Twin
41
Digital
Twin
Real-Time
Inventory
Management
Kafka Connect
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity
Storage Processing

Connected Car Infrastructure
42
https://www.youtube.com/watch?v=yGLKi3TMJv8
• Real Time Data Analysis
• Swarm Intelligence
• Collaboration with Partners
• Predictive AI
• …

Apache Kafka
Kafka as Digital Twin
43
Real-Time
Inventory
Management
Kafka Connect
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity
Digital Twin
Storage Processing

Construction site management
Collaborative planning
Inventory and asset management
Track, manage, and locate tools and equipment anytime and anywhere
https://www.confluent.io/customers/bosch/
https://events.confluent.io/online-talks/bosch-power-toolse-nables-real-time-analytics-on-iot-event-streams

Blockchain

Blockchain for Supply Chain Management
(Manufacturing, Logistics, …)
46
https://coletiv.com/blog/blockchain-supply-chain-management-system/
https://www.disruptordaily.com/blockchain-use-cases-supply-chain-management/

Real-time
High throughput
Decentralized database
Distributed log of records
Immutable log
Replication
High availability
Decoupling of applications / clients
Role-based access control to data
Tamper-proof
Encrypted payloads
Cross-company
is not a Blockchain!
X

Kafka AND Blockchain
48
Kafka
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk, etc.)
Blockchain
(Bitcoin)
Blockchain
(Ethereum)
Kafka-native
Blockchain
Kafka AS Blockchain
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk, etc.)

Kafka vs. Blockchain for Supply Chain Management
49
Use Tamper-proof
Apache Kafka for
• Enterprise infrastructure
• Open, scalable, real-time
requirements
• Flexible architectures for many use
cases
Use Hyperledger,
Ethereum, et al. for
• Deployment over various
independent organizations
• Participants verify the distributed
ledger contents themselves.
• Specific use cases
• Server-side managed and
controlled by multiple
organizations
• Scenarios where the business
value overturns the added
complexity and project risk
Use Kafka and
Blockchain together to
combine the benefits of
both
(where this makes sense!)
https://events.confluent.io/blockchain-kafka
https://www.kai-waehner.de/blog/2020/07/17/apache-kafka-blockchain-dlt-comparison-kafka-native-vs-hyperledger-ethereum-ripple-iota-libra/

The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...

Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE
COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based
Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance & Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete
Engagement Model
Revenue / Cost / Risk Impact
TCO / ROI
Executive Buyer

Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

Supply Chain Optimization with Apache Kafka

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Supply Chain Optimization with Apache Kafka

Similar to Supply Chain Optimization with Apache Kafka (20)

More from Kai Wähner

More from Kai Wähner (20)

Recently uploaded

Recently uploaded (20)

Supply Chain Optimization with Apache Kafka