Apache Kafka in Manufacturing and Industry 4.0

Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de
Event Streaming Use Cases and
Architectures in Manufacturing 4.0
Automotive, Aerospace, Semiconductors, Oil&Gas, Chemical, Food, and others…
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.deApache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de
1. Industry 4.0 - Digital Transformation for the Industrial IoT (IIoT)
2. 10000 Foot View – Event Streaming for Industry 4.0
3. Track&Trace / Production Control / Plant Logistics
4. Quality Assurance / Yield Management and Predictive Maintenance
5. Supply Chain Management
6. Cybersecurity
7. Servitization with Digital Twins
8. Additive Manufacturing
9. Augmented Reality
Agenda

Industry 4.0 / Industrial IoT (IIoT)

Digitalization changes the whole manufacturing process
https://www.mckinsey.com/industries/advanced-electronics/our-insights/reimagining-industrial-supply-chains

History of Automation Industry vs. Big Data and Cloud
Christofer Dutz (codecentric)
https://foss-backstage.de/sites/foss-backstage.de/files/2018-07/Revolutionizing%20Industrial%20IoT%20with%20Apache%20PLC4X.pdf
IIoT != IoT
Legacy / Proprietary
Monoliths
Long lifecycles
No Security

Evolution of Convergence between IT and Industrial Automation
https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence

Huge demand to build an open, flexible, scalable platform
• Real time
• Scalability
• High availability
• Decoupling
• Cost reduction
• Flexibility
• Elasticity
• Standards-based
• Extendibility
• Security
• Infrastructure-independent
• Multi-region / global

1. Industry 4.0 - Digital Transformation for the Industrial IoT (IIoT)
2. 10000 Foot View – Event Streaming for Industry 4.0
3. Track&Trace / Production Control / Plant Logistics
4. Quality Assurance / Yield Management and Predictive Maintenance
5. Supply Chain Management
6. Cybersecurity
7. Servitization
8. Additive Manufacturing
9. Augmented Reality
Agenda

STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming

An Event Streaming Platform
is the Underpinning of an Event-driven Architecture
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data warehouse
Producers
Consumers
Streams of real time events
Stream processing
apps
Connectors
Connectors
Stream processing
apps
Supplier
Alert
ForecastInventory Customer
Order

Hybrid Architecture - On Premise vs. Public Cloud
VM
SELF MANAGED
FULLY MANAGED
Edge vs. regional vs. global deployments
Cloud-first (greenfield) vs.
Hybrid architecture vs.
Strategic move from on premise to cloud

Digital Service at the Edge
Single
Kafka Broker
or Cluster
Digital
Service
Self-managed or
certified OEM Hardware
Kafka
Cluster
in DC /
Cloud
Replicator
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS

Centralized Digital Service
Digital
Service
Single
Kafka Broker
(or Cluster)
Self-managed or
Single
Kafka Broker
(or Cluster)
Self-managed or

Global Event Streaming
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO=0
Stream Data Globally with
Replication and Cluster Linking

Tesla
Trillions of messages per day for IoT use cases
https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/
https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/

Traceability and individual product serialization
Production control
Plant logistics
Minimization of waste
Elimination of fraud
Inventory transparency

Kafka Connect
Kafka Cluster
MQTT
Integration
Domain-Driven Design (DDD) for your Event Steaming Platform
SAP
Integration
Real Time
Predictions
IoT Platform
Connector
Java / Python /
”you-name-it”
Schema
Registry
Event Streaming Platform
IoT Domain OT Domain Analytics Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
22

More than just Messaging! Data Correlation in Real-Time…
https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006

Construction site management
Collaborative planning
Inventory and asset management
Track, manage, and locate tools and equipment anytime and anywhere
https://www.confluent.io/customers/bosch/
https://events.confluent.io/online-talks/bosch-power-toolse-nables-real-time-analytics-on-iot-event-streams

Food Value Chain
IoT-Based and Data-Driven
Single source of truth
across the food value chain
(in the factories, and across regions)
Business critical
operations
(tracking, calculations, alerts, …)
https://www.confluent.io/blog/creating-iot-based-data-driven-food-value-chain-with-confluent-cloud/

Location-Analytics and Geofencing
• Geofence: Virtual perimeter
for a real-world geographic
area
• Dynamically generated or
predefined set of boundaries
(such as plant area)
• Location-aware device of a
location-based service (LBS)
user entering or exiting a geo-
fence
• Trigger events / alerts
(enter, exist, outside, inside)
https://www.slideshare.net/gschmutz/location-analytics-realtime-geofencing-using-apache-kafka-203115651
https://developer.confluent.io/podcast/location-data-and-geofencing-with-apache-kafka-ft-guido-schmutz

Overall Equipment Effectiveness (OEE)
The gold standard for measuring manufacturing productivity
https://www.oee.com/oee-six-big-losses.html
OEE score 100%
=
100% Quality (only good parts)
+
100% Performance (as fast as possible)
+
100% Availability (no stop time)

Analytics
Database
Data Lake
CRM
Kafka as Data Historian
Continuous real time data ingestion, processing and monitoring 24/7 at scale
MQTT Modbus Firewall
Siemens S7
Kafka Connect
Integration | Storage | Processing | Access | Cloud | Security
Storage
Kafka Streams
/
ksqlDB
OPC-UA
Stateless +
Stateful
Kafka
Connect
Kafka
Clients
Java, C++. Python, Go, …
REST
Proxy
SCADA
DCS
ERP
MES
SyslogFile
X
Y
Z
A
B
C
Cloud
Factory

Reduction in Production Cost – Early Part Scrapping
Station 1 Station 2
Cost Before
100$
Total Cost
450$
(or more)
200$ 150$
Scrap? Scrap?

BI
Tool
AI/ML
Machine Vision for Quality Assurance and Yield Management
Apache Kafka and Applied Machine Learning
Filter, transform
aggregate, orchestrate
APP
Real-time alerting
Sensor Data
SCADA
MES
PLCs
OT
Team
Plant
Manager
Images
from Products
of Assembly Lines
IT
Team
Live
Ops
Machine Vision for
Quality Inspection
Reporting
Backup
Data Science Team
Data Lake

Supply Chain Management (SCM)
• Planning and coordination of all the people,
processes, and technology involved in creating
value for a company
• Cross cutting processes, including purchasing /
procurement, logistics, operations /
manufacturing, and others
• Automation, robustness, flexibility, hybrid
deployment (edge + cloud)
Six Sigma (6σ)

SCM
=
Zoo of
technologies
and products
Options:
No software
Buy
Rent
Make
Mix
TMS
Transport Management System
WMS
Warehouse Management System
WES
Warehouse Execution System
DPS
Demand Planning System
DRP
Distribution Requirements Planning
LMS
Labor Management System
CRM
Customer Relationship Management
SRM
Supplier Relationship Management
ERP
Enterprise
Resources
Planning
BI
Business Intelligence
MES
Manufacturing Execution System
PLM
Product Lifecycle Management

Postmodern ERP (coined by Gartner)
Replace legacy, monolithic and highly customized ERP suites
by a mixture of loosely coupled, exchangeable cloud-based and on-premises applications.
TMS
Legacy Proprietary
SOAP Web Services
Supplier
Alert
ForecastInventory Customer
Order
Core ERP
CRM
SaaS
Kafka Interface
MES
Proprietary
HTTP Web Services
LMS
Legacy Homegrown
Database + CDC
SRM
Kafka-native

Supply Chain Planning
Streaming and Batch Analytics Use Case
• Prepare for the unknown and the
unknowable
• Act at the right time in the right
context
• Yossi Sheffi: “You need to have
sensors in the ground”, then SCM
becomes a process of sensing and
responding
What if…?

From a batch-oriented systems architecture to
a streaming micro services platform
Thoughtworks:
https://www.confluent.io/kafka-summit-sf17/Fast-Data-in-Supply-Chain-Planning

Strangler Design Pattern
The Big Bang does not always work…
“The most important
reason to consider a
strangler fig application
over a cut-over rewrite
is reduced risk.”
Martin Fowler
Thoughtworks:
https://www.confluent.io/kafka-summit-sf17/Fast-Data-in-Supply-Chain-Planning

Supply Chain Purchasing
Real-time Natural Language Processing (NLP) for Digital Contract Intelligence

BMW Group
Industry-ready NLP Service Framework Based on Kafka
https://www.confluent.io/kafka-summit-lon19/industry-ready-nlp-service-framework-kafka/

Supply-chain operations reference (SCOR) model

Cross-Company Supply Chain Integration
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Cluster Linking
Tier 2
Supplier
OEM Streaming integration
between companies
API Management
(REST et al) is not
appropriate for
streaming data
Infosec and politics are
your biggest hurdle
Tier 1
Supplier

Kafka vs. Blockchain for Supply Chain Management
46
Use Tamper-proof
Apache Kafka for
• Enterprise infrastructure
• Open, scalable, real-time
requirements
• Flexible architectures for many
use cases
Use Hyperledger,
Ethereum, et al. for
• Deployment over various
independent organizations
• Participants verify the distributed
ledger contents themselves.
• Specific use cases
• Server-side managed and
controlled by multiple
organizations
• Scenarios where the business
value overturns the added
complexity and project risk
Use Kafka and
Blockchain together to
combine the benefits
of both
(where this makes sense!)
https://events.confluent.io/blockchain-kafka
https://www.kai-waehner.de/blog/2020/07/17/apache-kafka-blockchain-dlt-comparison-kafka-native-vs-hyperledger-ethereum-ripple-iota-libra/

https://news.microsoft.com/transform/hackers-hit-norsk-hydro-ransomware-company-responded-transparency/
https://www.insurancejournal.com/news/international/2019/04/30/525093.htm

Cybersecurity
The threat is real!
Challenges
Stealing IP
DDoS
Ransomware / wiperware
WannaCry, NotPetya, …
Damage: Billions of dollars
”Supply chain attack”
Industry 4.0
Networking
Communication
Connectivity
Open standards
”Always-on”

Legacy SIEM needs to evolve
ForwarderNetwork traffic
Firewall logs
RDBMS
Application logs
Adaptors
Beats
Sensor Data
Challenges:
● Proprietary forwarders that can only
send data to single source
● Data locked from being shared
● Difficult to scale with growing data
volumes
● Prohibitively high indexing costs
● Unable to filter out noisy data
● Slow batch processing
HTTP proxy logs

AI/ML
Modernized security information and event management (SIEM)
Filter,
transform,
aggregate
APP SIEM Index
Search
Curated streams
Forensic
Archive
HDFS
S3
Big Query
Syslog
CDC
Network traffic
Firewall logs
RDBMS
Application logs
Sensor Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic

Hybrid Cyber Data Platform
Secure intermediary (on Linux) between the existing (Windows) hardware and modern (Linux) infrastructure
Customer/Site 1 Customer/Site 2 Customer/Site N
Central Data Center / Public Cloud

Software and Digital Services become the Key Differentiator
54
https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery

Digital Twin – Merging the Physical and the Digital World
56
• Downtime reduction
• Inventory management
• Fleet management
• What-if simulations
• Operational planning
• Servitization
• Product development
• Healthcare
• Customer experience
“Virtual representation of something else (Physical thing, process, service)”
“A living model that drives a business outcome”
https://www.youtube.com/watch?v=Ri0TD7kYsIQ

Digital Thread
57
Digital Twin vs. Digital Thread?
I only use the term Digital Twin in the following slides.
Both terms overlap, often meaning the same.
Span
the
entire
lifecycle

Apache Kafka
Kafka as IoT Platform for the Digital Twin
58
Digital
Twin
Real
Time
App
Batch
App
Request
Response
App
Kafka Connect
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity
Storage Processing

Apache Kafka
Kafka as Digital Twin
59
Digital Twin
Real
Time
App
Batch
App
Request
Response
App
Kafka Connect
Storage Processing
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

Connected Car Infrastructure
61
https://www.youtube.com/watch?v=yGLKi3TMJv8
• Real Time Data Analysis
• Swarm Intelligence
• Collaboration with Partners
• Predictive AI
• …

Additive Manufacturing (aka 3D Printing)
https://www.researchgate.net/figure/Products-that-Prosumers-want-to-realize-with-3D-printing_tbl5_320243961
(A prosumer is a person who consumes and produces a product)
Prosumer
at Home
Smart
Factory

Distributed Manufacturing
• 3D printing everything from houses and human organs to parts for the Mars
Rover and even pizza
• Game changer for supply chains
• Continuous streaming and process analytics
• Design files are intellectual property (and can easily be replicated)
• Streaming design files
• transforms how we consume physical products
• secures illegal use
https://www.computerworld.com/article/2836714/how-3d-printing-and-the-netflix-model-are-changing-everything.html
https://spaceflight101.com/iss/made-in-space-1st-iss-3d-printer

Additive Manufacturing (aka 3D Printing)
Use Cases for Event Streaming with Apache Kafka
Production
Continuous streaming and process analytics
for cost reduction / quality improvement
Customer 360
Customer care, sales, support
for full product lifecycle
Security
Securing intellectual property
through a streaming model
instead of file sharing

Augmented Reality in Manufacturing
Hand holding mobile smartphone use AR application
to check and control welding robotics automatic arms machine
Virtual Prototyping
Smart Assistance
Logistics &
Warehouse Management

Augmented Reality (AR) Tools and SDKs
AR SDKs
Vuforia, Wikitude, ARKit,
ARCore, MaxST, EasyAR,
Kudan, ARToolKit, Armedia,
DeepAR.ai
Virtual Dimension Center (VDC) Fellbach

Augmented Reality for Smart Assistence
with Apache Kafka, Kafka Connect and ksqlDB
Pre-Processing and Data Correlation
(Kafka Streams / ksqlDB)
Receive
Command
Operator
(REST Proxy)
MES
(Java)
Send
Live
Metrics
Send
Command
Send
Production
StatusRobots
(C++)
Receive
Correlated
Information

Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF
CHOICE
COMMITTER-DRIVEN
EXPERTISE
PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based
Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete
Engagement Model
Revenue / Cost / Risk
Impact
TCO / ROI
Executive Buyer

I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid, Multi-
Region)
Build Contextual Event-
Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...

Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

Apache Kafka in Manufacturing and Industry 4.0

Recommended

Recommended

More Related Content

More from Kai Wähner

More from Kai Wähner (20)

Recently uploaded

Recently uploaded (20)

Apache Kafka in Manufacturing and Industry 4.0