Apache Kafka for Industrial IoT and Manufacturing 4.0

Event Streaming for Industrial IoT
and Manufacturing 4.0
Use Cases and Architectures for Data in Motion powered by Apache Kafka
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de

Industrial IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Industry Trends – The World Gets Connected
2
People on the Internet

Industry Trends - Software and Digital Services become the Key Differentiator
3
https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery

This is a fundamental paradigm shift...
5
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Apache Kafka is the Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time Alerting
System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
6

MQTT
Data
Historian
STREAM
PROCESSING
CONNECTORS
Example Architecture for Event Streaming
ksqlDB
KStreams
Bi-directional real-time integration between the OT and IT worlds
APP
Oracle
DB
ORACLE CDC
SOURCE
CONNECTOR
APP

Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital replatforming/
Mainframe Offload
Connected Car: Navigation & improved in-
car experience: Audi
Customer 360
Simplifying Omni-channel Retail at Scale:
Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives; LinkedIn,
Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka Streams:
Funding Circle
Detect Fraud & Prevent Fraud in Real Time:
PayPal
Kafka as a Service - A Tale of Security and
Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)

10 Reasons for Event Streaming with Apache Kafka
Real Time
Scalable
Cost Reduction
24/7 – Zero downtime, zero data loss
Decoupling – Storage, Domain-driven Design
Data (re-)processing and stateful client applications
Integration – Connectivity to IoT, legacy, big data, everything
Hybrid Architecture – On Premises, multi cloud, edge computing
Fully managed cloud
No vendor locking
9

Cloud-Native Deployment leveraging Kubernetes
10

Truly CLOUD-NATIVE experience
at the edge, in the data center, and in the cloud
11
Confluent Cloud
A fully managed, cloud-native service for Apache Kafka®
Confluent Platform
A complete, enterprise-grade distribution of Apache Kafka®
Confluent for
Kubernetes
Ansible
Playbooks
Packages: Docker,
RPMs, Tarball
Public Cloud Workloads Edge and On-Premise Workloads
On Kubernetes On VMs / Bare Metal

Shipping Industry
Marine, Oil Transport, Vessel Fleet, Shipping Line
Real-time Operations, Logistics, Predictive Maintenance, Security
12
Customer Data
Crew, Cargo
Vessel Data
Fuel Consumption, Speed,
Planned Maintenance
Automatic Identification System (AIS)
Unique Identification,
Position, Course, Weather, Draft
Drone Data
Deliveries,
Survey/Inspection
of Assets such as Oil Rigs,
Pipelines, Offshore Turbines
Edge Analytics
Bidirectional Edge to Cloud Integration
Data Ingestion
Stream
Processing
Data
Integration
Logistics
Track&Trace
Routing
Monitoring
Alerting
Command&Control
Batch Analytics
Reporting
Machine Learning
Backend Systems
Oracle, SAP, OSIsoft
PI, etc.
X = Event Streaming
X = Other Technologies
Bi-Directional Hybrid Cloud
Replication

Kafka is “Real-Time”, but NOT “Hard Real-Time”
OT - Connected Vehicle (Car,
Train, Drone)
OT - Manufacturing
(Field Bus, PLC, Machine, Robot)
IT – Enterprise Software
(Data Center, Cloud, Car IT)
Central Data Center / Public Cloud
Connect
Vehicle Data (e.g. OBD2)
Robot Data All Data
C
C++
C
C++
Java
Python
Go
[#] Hard Real Time
= Deterministic network
with zero spikes + zero latency
[#] Soft Real Time
+ Near Real Time
+ Batch

Device management
Unreliable networks
Connectivity beyond standards
Tens of thousands of connections
Embedded systems
…
is not an IoT Platform!

Severstal
Predictive Maintenance and Quality Assurance at the Shop Floor
Real Time Streaming Machine Learning with Kafka
https://www.confluent.io/customers/severstal/

BMW
Decoupled Logistics and Manufacturing
Mission-critical workloads at the edge and in the cloud
• Why Kafka? Decoupling. Transparency. Innovation.
• Why Confluent? Stability is key in manufacturing
• Decoupling between logistics and production systems
• Provide edge platform (self-managed) + Azure Cloud (fully-managed) +
bidirectional integration
• Use case
• Logistics and supply chain in global plants
• Right stock in place (physically and in ERP systems like SAP)
• Just in time, just in sequence
• Lot of critical applications
• Things BMW couldn’t do before
• Get IoT data (without interfereing with others), get it to the right place
• Collect once, process and consume several times (at different times)
• Enable scalable real-time processing and improve time-to-market with
new applications
17
Jay Kreps, Confluent CEO
Felix Böhm, BMW Plant Digitalization and Cloud Transformation
Keynote at Kafka Summit EU 2021:
https://www.youtube.com/watch?v=3cG2ud7TRs4
(My Notes from the BMW Keynote at Kafka Summit EU 2021)

Food Value Chain
IoT-Based and Data-Driven
Single source of truth
across the food value chain
(in the factories, and across regions)
Business critical operations
(tracking, calculations, alerts, …)
https://www.confluent.io/blog/creating-iot-based-data-driven-food-value-chain-with-confluent-cloud/

Track, manage, and locate tools and
other equipment
anytime and anywhere from the
warehouse to the jobsite https://www.confluent.io/customers/bosch/
https://events.confluent.io/online-talks/bosch-power-toolse-nables-real-time-analytics-on-iot-event-streams

Devon Energy
Oil & Gas Industry
Improve drilling and well completion operations
Edge stream processing/analytics + closed-loop control ready
Vendor agnostic (pumping, wireline, coil, offset wells, drilling
operations, producing wells)
Replication to the cloud in real-time at scale
Cloud agnostic (AWS, GCP, Azure)
Source: Energy in Data - Powered by AAPG, SEG & SPE: energyindata.org

‘My Porsche’
A digital service platform for customers, fans, and enthusiasts
21
https://medium.com/porschedev

Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html

Edge, Hybrid, Global
IoT Architectures

Analytics
Database
Data Lake
CRM
Kafka as Data Historian
Continuous real time data ingestion, processing and monitoring 24/7 at scale
MQTT Modbus Firewall
Siemens S7
Kafka Connect
Integration | Storage | Processing | Access | Cloud | Security
Storage
Kafka Streams
/
ksqlDB
OPC-UA
Stateless + Stateful
Kafka
Connect
Kafka Clients
Java, C++. Python, Go, …
REST
Proxy
SCADA
DCS
ERP
MES
Syslog
File
X
Y
Z
A
B
C
Cloud
Factory

Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint Edge
Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
for RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
25

Postmodern Real-Time Locating / Tracking System (RTLS)
C2
Time
P
C3
C1
C2
C3
Asset / People Tracking
Real-Time Alerting
Geofencing for
Distance Enforcement
Continuous
Real-Time Aggregation
Plan Future Location
Requirements
Batch Analytics
on Historical Events

CRM
Real-Time Location System (RTLS)
for Asset Tracking
Customer data
Advanced Planning
and Scheduling (APS)
Manager
Get report
API
Customer Customer
Customer
data
Truck
schedule
Payment
data
Route
details
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Customer
data
Train
schedule
Payment
data
Loyalty
information
Postmodern Asset Tracking
Wavelength
Public Cloud VPC
Campus 5G Telco Carrier 5G

Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed Commit
Log
Streaming Machine Learning (Training + Inference)
with Kafka, Tiered Storage and TensorFlow IO
https://github.com/tensorflow/io
28
Model X
(at a later time)

Confluent Tiered Storage for Kafka
Object Store
Processing Storage
Transactions,
auth, quota
enforcement,
compaction, ...
Local
Remote
Kafka
Apps
Store Forever
Older data is offloaded to inexpensive object storage,
permitting it to be consumed at any time.
Save $$$
Storage limitations, like capacity and duration, are
effectively uncapped.
Instantaneously scale up and down
Your Kafka clusters will be able to automatically self-balance
load and hence elastically scale
(Only available in Confluent Platform)

IoT Infrastructure with
MQTT, Kafka, TensorFlow and MongoDB
30
MQTT
Proxy
Kafka
Cluster
Kafka
Connect
Car Sensors
Kafka Ecosystem
TensorFlow
Other Components
Kafka
Streams
(Java)
All
Data
Critical
Data
Ingest
Data
Potential
Detect
KSQL
TensorFlow
Train
Analytic Model
Consume
Data
Preprocess
Data
Analytic Model
Deploy
Analytic
Model
Python
MongoDB
Storage
MongoDB
Dashboards
Search
Analytics
Mobile App
BI Tool
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

Kafka Connect
Kafka Cluster
CRM Integration
Domain-Driven Design for your Integration Layer
OT
Integration
Custom
Application
OSIsoft PI
Java / KSQL /
Kafka Streams
Schema Registry
Event Streaming Platform
Customer
Domain
OT
Domain
Asset Management
Domain
è Independent and loosely coupled, but scalable, highly available and reliable!

Cross-Company Streaming Data Exchange
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Cluster Linking
Tier 1 Mobility
Service
Streaming integration
between companies
API Management
(REST et al) is not appropriate
for streaming data
Infosec and politics are your
biggest hurdle
OEM

AI/ML
Modernized security information and event management (SIEM)
Filter, transform,
aggregate
APP SIEM Index
Search
Curated streams
Forensic
Archive
HDFS
S3
Big Query
Syslog
CDC
Network traffic
Firewall logs
RDBMS
Application logs
Sensor Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic

Event Streaming in Air-Gapped and Zero Trust Environments
Secure intermediary (on Linux) between the existing (Windows) hardware and modern (Linux) infrastructure
Customer/Site 1 Customer/Site 2 Customer/Site N
Central Data Center / Public Cloud

Confluent Data Diode
Software-based Unidirectional Gateway for Zero Trust Security Architectures
Streaming from Industrial Networks to Enterprise Networks
UDP-based Source and Sink Kafka Connectors for High Volume and Open Architecture
Run over a one-way/UDF hardware interface
https://docs.confluent.io/kafka-connect-data-diode

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual Event-
Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
42

The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
43

Car Engine Car Self-driving Car
Confluent Completes Apache Kafka

Confluent... Complete. Cloud-native. Everywhere.
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage Prerequisites
Partnership for Business
Success

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Apache Kafka for Industrial IoT and Manufacturing 4.0

Recommended

Recommended

More Related Content

More from Kai Wähner

More from Kai Wähner (20)

Recently uploaded

Recently uploaded (20)

Apache Kafka for Industrial IoT and Manufacturing 4.0