Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-machine-learning-for-supply-chain Automating multifaceted, complex workflows requires hybrid solutions like streaming analytics of IoT data, batch analytics like machine learning solutions, and real-time visualizations. Leaders in organizations who are responsible for global supply chain planning are responsible for working with and integrating with data from disparate sources around the world. Many of these data sources output information in real-time, which assists planners in operationalizing plans and interacting with manufacturing output. IoT sensors on manufacturing equipment and inventory control systems feed real-time processing pipelines to match actual production figures against planned schedules to calculate yield efficiency.  Using information from both real-time systems and batch optimization, supply chain managers are able to economize operations and automate tedious inventory and manufacturing accounting processes. Sitting on top of all these systems is a supply chain visualization tool, enabling users' visibility over the global supply chain. If you are responsible for key data integration initiatives, join for a detailed walk through of a customer's use of this system built using Confluent and Expero tools.  WHAT YOU'LL LEARN:  • See different use cases in automation industry and Industrial IoT (IIoT) where an event streaming platform adds business value.  • Understand different architecture options to leverage Apache Kafka and Confluent. • How to leverage different analytics tools and machine learning frameworks in a flexible and scalable way. • How real-time visualization ties together streaming and batch analytics for business users, interpreters, and analysts. • Understand how streaming and batch analytics optimize the supply chain planning workflow. • Conceptualize the intersection between resource utilization and manufacturing assets with long term planning and supply chain optimization.

1. 1Apache Kafka and Machine Learning – Kai Waehner
Industrial Internet of Things (IIoT) at Scale
Real Time Planning Optimization and Predictive Maintenance
with an Event Streaming Platform
Kai Waehner
kai.waehner@confluent.io
@KaiWaehner
Graham Ganssle, Ph.D.
graham@experoinc.com
@GrahamGanssle

2. Confluents - Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔

3. 3
Connected Intelligence (Cars, Machines, Robots, …)

4. 4
Smart Cities

5. 5
Smart Retail and Customer 360

6. 6
Intelligent Applications (Early Part Scrapping, Predictive Maintenance, …)

7. 7
Business Digitalization Trends are Driving the Need to Process
Events at a whole new Scale, Speed and Efficiency
The world has changed!

8. 8Best-of-breed Platforms, Partners and Services for Multi-cloud Streams
Private Cloud
Deploy on bare-metal, VMs,
containers or Kubernetes in your
datacenter with Confluent Platform
and Confluent Operator
Public Cloud
Implement self-managed in the public
cloud or adopt a fully managed service
with Confluent Cloud
Hybrid Cloud
Build a persistent bridge between
datacenter and cloud with
Confluent Replicator
Confluent
Replicator
VM
SELF MANAGED FULLY MANAGED

9. Data Lake
Batch
Analytics
Event
Streaming
Platform
Batch
Integration
Real Time Pre-
processing
Machine Sensors
Streaming Platform
Other Components
Real Time
Processing
(6b) All Data
(3)
Read Data
Optimization
/ Analytics
(5)
Deploy
Optimization
Model
(2)
Preprocess
Data
Model
Standard
based
Integration
(8a)
Stop Machine
(1)
Ingest Data
Real Time Edge
Computing
Model Lite
Real Time App
Model Server
RPC
PLC Proprietary
based
Integration
Standard
Interface
Proprietary
Interface

10. Spark
Notebooks
(Jupyter)
Kafka
Cluster
Kafka
Connect
KSQL
Machine Sensors
Kafka Ecosystem
Other Components Real Time
Kafka Streams
Application
(Java / Scala)
(6b) All Data
(3)
Read Data
TensorFlow I/O
TensorFlow
(5)
Deploy Model
(2)
Preprocess
Data
TensorFlow
MQTT
File
HTTP
(8a)
Stop Machine
(1)
Ingest Data
Real Time Edge
Computing
(C / librdkafka)
TensorFlow Lite
Real Time Kafka
App
TensorFlow
Serving
HTTP /
gRPC
(4)
Train Model
PLC
Beckhoff
S7
Modbus
OPC-UA
PLC4X
Connector
Kafka Connect
Standard
Interface
Proprietary
Interface

11. 11
Confluent Platform
The Event Streaming Platform Built by the Original Creators of Apache Kafka®
Operations and Security
Development & Stream Processing
Apache Kafka
Confluent Platform
Mission-Critical Reliability
Complete Event
Streaming Platform
Freedom of Choice
Datacenter Public Cloud Confluent Cloud
Self-Managed Software Fully Managed Service

12. 12
Confluent Platform Licensing
Open Source features
Apache Kafka®
Apache 2.0 License
Free. Unlimited Kafka brokers
Community support
Enterprise License (paid)
● Annual subscription
● 24x7 Confluent support
● Kafka Connect
● Kafka Streams
Apache ZooKeeper™
Clients
Ansible Playbooks
Community features
Connectors
Confluent Community License
Free. Unlimited Kafka brokers
Community support
REST Proxy
KSQL
Schema Registry
Commercial features
Connectors
Developer License
● Free
● Limited to 1 Kafka broker
● Community support
Evaluation License
● Free 30-day trial
● Unlimited Kafka brokers
● Community support
Control Center
Command Line Interface
Replicator
Auto Data Balancer
MQTT Proxy
Operator
Security Plugins
Role-Based Access Control (preview) ● Best-effort Confluent Support
New in CP 5.3

13. 1313
Confluent Operator:
Apache Kafka on
Kubernetes made
simple
Run Apache Kafka and Confluent
Platform as a cloud-native application
on Kubernetes to minimize operating
complexity and increase developer
agility
Confluent Platform
Kubernetes
AWS Azure GCP
RH OpenShift Pivotal
On-Premises Cloud
Docker Images
Confluent Operator

14. 1414
Confluent
Operator
Deploy to Production in
Minutes
Automated deployment of
Confluent Platform resources:
Brokers, ZooKeeper, Kafka Connect,
KSQL, Schema Registry, Control
Center, and Replicator
Automate Key Lifecycle
Operations
● Failover
● Automated rolling upgrades
● Elastic scalability
Deploy on Any Platform,
On-Prem or in the Cloud
Run at Scale with
Confidence
Operationalizes years of Confluent
Cloud experience into a proven,
enterprise-grade solution that you
can deploy without deep Kafka
expertise
Deploy Apache Kafka
and Confluent Platform
as a cloud-native system
on Kubernetes
Kubernetes Engine Elastic Container
Service for Kubernetes
Kubernetes Service
https://www.slideshare.net/KaiWaehner/c
onfluent-operator-as-cloudnative-kafka-
operator-for-kubernetes

15. Confluent Cloud
Cloud-Native Confluent Platform Fully-Managed Service
Available on the leading public clouds with mission-critical SLAs.
Serverless Kafka characteristics:
Pay-as-you-go, elastic auto-scaling, abstracting infrastructure (topics not brokers)

16. Confluent Cloud, What does Fully-managed Mean?
Infrastructure
management
(commodity)
Scaling
● Upgrades (latest stable version of Kafka)
● Patching
● Maintenance
● Sizing (retention, latency, throughput, storage, etc.)
● Data balancing for optimal performance
● Performance tuning for real-time and latency requirements
● Fixing Kafka bugs
● Uptime monitoring and proactive remediation of issues
● Recovery support from data corruption
● Scaling the cluster as needed
● Data balancing the cluster as nodes are added
● Support for any Kafka issue with less than 60 minute response time
Infra-as-a-Service
Harness full power of Kafka
Kafka-specific
management
Platform-as-a-Service
Evolve as you
need
Future-proof
Mission-critical reliability
Most Kafka as a Service offerings are partially-managed

17. WE BRING CHALLENGING IDEAS TO REALITY
Data Science & Machine Learning
ML ops, devops, analytics integration
Rapid Prototypes (UX, Data & ML)
User Experience & Data Visualization
Full Stack Software Architecture & Development
Product Innovation
Graph Data Modeling & Visualization
Product Assessments, Roadmaps & Selection
Training
17

18. © 2019 Expero, Inc. All Rights Reserved
18
Biotech
Semiconductors Financial Services
Software Supply Chain
Defense & Justice 18

19. Expero’s main offices are in
Houston and Austin, Texas.
Delivery teams include expert staff
from around the United States,
Canada, Spain, Argentina, and
Romania. We make software for
clients in the US, Europe, Australia
and Japan.
19

20. © 2019 Expero, Inc. All Rights Reserved
Challenges
● Overwhelming Options
● Complex Alternatives
● Endless Dependencies
● Large Data Sets
● Disconnected Systems
● “Good Enough?!?”
‘Overwhelming the Human’ with Data
20

21. © 2019 Expero, Inc. All Rights Reserved
Complex Alternatives
The Plan Is Never Perfect
● The given plan is inaccurate, how do these
two human-designed options compare?
● Can I get a decent answer now instead of a
perfect answer tomorrow?
● A human can often out-think an optimizer, if
she has help visualizing constraints her
ideas break
21

22. © 2019 Expero, Inc. All Rights Reserved
Endless Dependencies
Exceptions are the Rule
● Which orders are affected by this incident?
● If this inventory shortage persists, which
manufacturing activity is at risk?
● What is the earliest I can get this package
to the final customer location?
● Which customer’s orders do I need to cut if
production or transportation falls short?
22

23. © 2019 Expero, Inc. All Rights Reserved
Large, Siloed Data
The World Keeps Happening
● Master data is fairly small
○ Routes, recipes, trucks, parts
● Transactional data is huge
○ Orders, shipments, scans
● Important data resides in multiple systems
AND spreadsheets
● Years of historical data
● A system that handles large write load while
allowing rapid access to the data is critical
for real time decision making
23

24. © 2019 Expero, Inc. All Rights Reserved
Streaming and Batch
Analytics Use Case: Supply
Chain Planning
24

25. Planners
forecast long
term schedule
Production
begins
IOT data from
production:
inventories,
manufacturing
machines,
yield metrics
Production
forecast
Forecasted
production -
plan diffs
Re optimize
plan based on
actuals
Change orders
to supply
chain:
inventory,
manufacturing
schedules
Change
operational
characteristics
: plant 223
needs new Al
extruder
Customer
delivery SLAs:
actuals vs.
plan
streaming analytics using Confluent
batch analytics using Expero ML
physical operations
miranda
viz
miranda
viz
miranda
viz
miranda
viz

26. © 2019 Expero, Inc. All Rights Reserved
Demo

27. Planners
forecast long
term schedule
Production
begins
IOT data from
production:
inventories,
manufacturing
machines,
yield metrics
Production
forecast
Forecasted
production -
plan diffs
Re optimize
plan based on
actuals
Change orders
to supply
chain:
inventory,
manufacturing
schedules
Change
operational
characteristics
: plant 223
needs new Al
extruder
Customer
delivery SLAs:
actuals vs.
plan
miranda
viz
miranda
viz
miranda
viz
miranda
viz
PLC4X
Connector
Kafka
ConnectMQTT
File
HTTP
Machine
Sensors
Kafka
Cluster
KSQL
Tensor
Flow
Kafka
Connect
Notebooks
(Jupyter)
Spark
Real Time
Kafka
App
streaming analytics using Confluent
batch analytics using Expero ML
physical operations
TensorFlow
Serving

28. SOLUTIONS
FOR SUPPLY CHAIN
Expero Miranda:
Solution Includes: UI, Data Model,
Backend Code, Platform Elements:
Graph, Time-series, Streaming,
NoSQL, Search & Analytics
SOLUTION : Supply Chain
● Inventory
● Planning
● Analytics
● Security
Extendable ML Architecture:
Risk, Planning, Analytics

29. © 2018 Expero, Inc. All Rights Reserved
Combination Dashboards:
• What If - Analysis
• Optimization - Outcome Analysis
29

30. © 2019 Expero, Inc. All Rights Reserved
Fulfilment Chain
Optimization
30
● Graph-based information propagation
● Temporal forecasting
● Optimal DC locations picker
● Inventory distribution
● Bin packing
● Delivery scheduling
● Flow simulation
● Dynamic routing
● MLOps

31. © 2019 Expero, Inc. All Rights Reserved
Inventory Balance
31

32. © 2019 Expero, Inc. All Rights Reserved
Business - Technical Match
● Craft a technology solution which solves a business
problem
Partner with Leading Software
● Confluent is a preferred partner in the streaming space
Delivered Functionality
● Iterate with stakeholders to ensure solution fit
6 - 8 Week PoC
● Rapid delivery of business value
32
Engagement Model

33. © 2018 Expero, Inc. All Rights Reserved
33
Foundation Session 2-3 Days
USABILITYUSEFULNESS
CONTENT
INTERACTION DESIGN
INFO ARCHITECTURE
FUNCTIONALITY
USER AUDIENCE
WHO are present and future users?
What are their goals?
WHAT functionality to keep?
WHAT to add?
HOW can UI frameworks and patterns
be employed to scale UX?
Homogenizing visual design & brand
Does terminology align with domain?
VISUAL DESIGN
Foundational
in creating
good UX
Beauty is skin deep | UX extends to Foundation

34. © 2019 Expero, Inc. All Rights Reserved
34
PROCESS FOR SUCCESS SUCCESSEDUCATE&ADVISECOLLABORATE
DELIVERMAP
MAP SOLUTION TO YOUR ENVIRONMENT
ALIGNED TO YOUR TECHNICAL NEEDS AND
PRIORITIZE USE CASES THAT ENABLE YOUR
DESIRED BUSINESS OUTCOMES
LIVE DEMO / DELIVER A SOLUTION
PROPOSAL THAT PROVIDES BEST PRACTICE
RECOMMENDATION, BUSINESS VALUE, AND
DEPLOYMENT SUCCESS PLAN
COLLABORATE THROUGH STRUCTURED
BUSINESS AND TECHNICAL DISCOVERY TO
ACCELERATE ALIGNMENT ACROSS
REQUIREMENTS, SUCCESS CRITERIA &
ROADMAP
EDUCATE & ADVISE HOW EXPERO HAS
PARTNERED WITH OTHER CUSTOMERS
ACROSS USE CASES TO DELIVER BUSINESS
AND TECHNICAL VALUE

35. © 2019 Expero, Inc. All Rights Reserved
35
Working Prototype : Engage Business Users Early
35
Prototype &
Test
Data DiscoveryState the Problem
& Zero in on User
Goals
Determine the Most
Effective
Presentation
PLAY: Rapid Pilot

36. © 2019 Expero, Inc. All Rights Reserved
36
PoC : Weekly Timeline ~6 - 8 Weeks
(Ex: Discovery & Development Stage)
Foundation FinalizeDevelopment Refinement
Detailed Design
PoC Sprint 1
First Review S1 Review S2 Demo Final Review
PoC Sprint 2 Final Sprint
* Full Team Standups
- Jira Board Features List
- Weekly Review per Sprint
PoC Sprint 3

37. 37Apache Kafka and Machine Learning – Kai Waehner
Questions? Feedback?Let’s connect!
Kai Waehner
kai.waehner@confluent.io
@KaiWaehner
Graham Ganssle, Ph.D.
graham@experoinc.com
@GrahamGanssle
https://experoinc.zoom.us/j/6549904076

38. Code: Exper20KS19
20% DISCOUNT*
*Standard Priced Conference pass