Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Machine Learning

A digital twin is a digital replica of a living or non-living physical entity. This session discusses the benefits and IoT architectures of a Digital Twin in Industrial IoT (IIoT) and its relation to Apache Kafka, IoT frameworks and Machine Learning. Kafka is often used as central event streaming platform to build a scalable and reliable digital twin for real time streaming sensor data. A live demo shows a scalable digital twin infrastructure for condition monitoring and predictive maintenance in real time for a connected car infrastructure leveraging Kafka, MQTT and TensorFlow.

Key Take-Aways:
• Learn about use cases and characteristics of a digital twin in various industries
• Understand how to build a digital twin for every single (of tens of thousands) IoT device or machine
• See different IoT architectures with Kafka and other IoT technologies and products, including edge, hybrid and global deployments
• Understand the relation to Machine Learning and bring added value to your IoT infrastructure by enabling use cases like predictive maintenance
• Understand how the Apache Kafka enables scalable and flexible end-to-end integration processing from IIoT data to various backend applications
• Watch a live demo of an end-to-end integration, real time processing and analytics of thousands of IoT devices

More details:

https://www.kai-waehner.de/blog/2019/11/28/apache-kafka-industrial-iot-iiot-build-an-open-scalable-reliable-digital-twin/

https://www.kai-waehner.de/blog/2020/03/25/architectures-digital-twin-digital-thread-apache-kafka-iot-platforms-machine-learning/ ‎

https://youtu.be/Q3eKPEVwNVY

  • Be the first to comment

IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Machine Learning

  1. 1. IoT Architectures for a Digital Twin with Apache Kafka and Event Streaming A Digital Replica of Things - Open, Scalable and Reliable Kai Waehner Technology Evangelist contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
  2. 2. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Abstract IoT Architectures for a Digital Twin with Apache Kafka and Event Streaming A digital twin is a digital replica of a living or non-living physical entity. This session discusses the benefits and IoT architectures of a Digital Twin in Industrial IoT (IIoT) and its relation to Apache Kafka and other IoT frameworks. Kafka is often used as central event streaming platform to build a scalable and reliable digital twin for real time streaming sensor data. A live demo shows a scalable digital twin infrastructure for condition monitoring and predictive maintenance in real time for a connected car infrastructure leveraging Kafka, MQTT and TensorFlow. Key Take-Aways: ● Learn about use cases and characteristics of a digital twin in various industries ● Understand how to build a digital twin for every single (of tens of thousands) IoT device or machine ● See different IoT architectures with Kafka and other IoT technologies and products ● Bring added value to your IoT infrastructure by enabling use cases like predictive maintenance ● Understand how the Apache Kafka enables scalable and flexible end-to-end integration processing from IIoT data to various backend applications ● Watch a live demo of an end-to-end integration, real time processing and analytics of thousands of IoT devices
  3. 3. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Key Takeaways • A Digital Twin merges the physical and the digital world • Apache Kafka enables an open, scalable and reliable infrastructure for a Digital Twin • Event Streaming complements IoT platforms and other backend applications / databases. + 3
  4. 4. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • IoT Platforms • Apache Kafka as Event Streaming Solution for IoT • Spoilt for Choice for a Digital Twin • Global IoT Architectures • A Digital Twin for 100000 Connected Cars 4
  5. 5. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • IoT Platforms • Apache Kafka as Event Streaming Solution for IoT • Spoilt for Choice for a Digital Twin • Global IoT Architectures • A Digital Twin for 100000 Connected Cars 5
  6. 6. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Software and Digital Services become the Key Differentiator 6 https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery
  7. 7. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Twin – Merging the Physical and the Digital World 7 • Downtime reduction • Inventory management • Fleet management • What-if simulations • Operational planning • Servitization • Product development • Healthcare • Customer experience “Virtual representation of something else (Physical thing, process, service)” “A living model that drives a business outcome” https://www.youtube.com/watch?v=Ri0TD7kYsIQ
  8. 8. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Granularity of Digital Twins 8 https://www.youtube.com/watch?v=cfbKR48nSyQ Remaining Useful Life
  9. 9. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Thread 9 Digital Twin vs. Digital Thread? I only use the term Digital Twin in the following slides. Both terms overlap, often meaning the same. Span the entire lifecycle
  10. 10. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Virtual Singapore: A Digital Twin of the (Smart) City 10 Design, Monitor and Manage Cities • Urban Planning (e.g. Crowd Simulation) • Collaboration and Decision-Making • Communication and Visualisation • Improved Accessibility • Analysis on Potential for Solar Energy Production • … https://www.nrf.gov.sg/programmes/virtual-singapore
  11. 11. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Smart Infrastructure: Digital Solutions for Entire Building Lifecycle 11 https://new.siemens.com/global/en/products/buildings/digitalization/digital-building-lifecycle.html • Safer, more secure, more efficient and resilient buildings • Continuously interaction, learning and adaption to create environments that care • Follow the entire digital building lifecycle
  12. 12. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Connected Car Infrastructure 12 https://www.youtube.com/watch?v=yGLKi3TMJv8 • Real Time Data Analysis • Swarm Intelligence • Collaboration with Partners • Predictive AI • …
  13. 13. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Twinning the Human Body to Enhance Medical Care 13 • Monitoring and evaluation without being in close proximity • Testing the impact of changes on the performance of a system • Smart machines are more advanced than humans • Determine what actions to take • Modelling an individual’s genomic makeup, physiological characteristics, and lifestyle to create personalized medicine • Capturing the human body will have multiple benefits for doctors such as discovering undeveloped illnesses, experimenting with treatments, and improving preparation for surgeries https://www.challenge.org/insights/digital-twin-in-healthcare/ https://youtu.be/H6JzPCbyVSM
  14. 14. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Market 2020 14 https://www.youtube.com/watch?v=w4TPeVO3dwo (Q1, 2020 Webinar IoT Analytics Research)
  15. 15. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Twin and Artificial Intelligence (AI) / Machine Learning • Complementary Concepts • Continuous Learning, Monitoring and Acting • (Good) data is key for success 15 https://towardsdatascience.com/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b
  16. 16. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Twin Applied… 16 https://www.youtube.com/watch?v=cfbKR48nSyQ
  17. 17. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Twin Applied… 17 https://www.youtube.com/watch?v=cfbKR48nSyQ
  18. 18. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Twin Applied… 18 https://www.youtube.com/watch?v=cfbKR48nSyQ
  19. 19. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Twin Applied… 19 https://www.youtube.com/watch?v=cfbKR48nSyQ
  20. 20. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • IoT Platforms • Apache Kafka as Event Streaming Solution for IoT • Spoilt for Choice for a Digital Twin • Global IoT Architectures • A Digital Twin for 100000 Connected Cars 20
  21. 21. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de History of Automation Industry vs. Big Data and Cloud https://foss-backstage.de/sites/foss-backstage.de/files/2018-07/Revolutionizing%20Industrial%20IoT%20with%20Apache%20PLC4X.pdf
  22. 22. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Challenges in Automation Industry IoT != IIoT != Buildings != Healthcare … • IoT = Connected cars, smart home, … à Large scale, secure, scalable, open, modern technologies • IIoT / Buildings = Not connected at all or slow, insecure, not scalable, proprietary • Healthcare = Often not connected at all yet (huge security requirements) Legacy / Proprietary IIoT Technologies • Usually incompatible protocols, typically proprietary • Usually serial connections (very low latency, nanoseconds) - with TCP / UDP wrapper around it to integrate with “external world” • Siemens S7, Modbus, Beckhoff, Profinet, Allen Bradley, etc. • OPC-UA (required machine update + license cost) Product Lifecycles • Long lifecycle (tens of years) • Factories and buildings cost millions, no simple changes / upgrades • Still using Windows 7 without Service Packs => Usability and security issues • Mantra: “Stay with your well-known vendor forever”
  23. 23. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Challenges in Automation Industry Monoliths • No scalability • No extendibility • No real failover (start your backup machine) Missing Security Capabilities • Security in software development == Authentication, Authorization, Antivirus, SSL, SASL, Kerberos • Security in automation industry == Safety • “if you press the red button, the machine stops immediately” • Insecure by nature => No Authentication / Authorization / Encryption • Mantra: “Our factory building and network is secure, no access from outside” • Contradicts with “move to cloud and big data analytics”
  24. 24. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Trends: Evolution of Convergence between IT and Industrial Automation https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence
  25. 25. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Complexity, Cost and Scalability are Main Blockers 25
  26. 26. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Huge demand to build an open, flexible, scalable platform • Real time • Scalability • High availability • Decoupling • Cost reduction • Flexibility • Standards-based • Extendibility • Security • Infrastructure-independent • Multi-region / global
  27. 27. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • IoT Platforms • Apache Kafka as Event Streaming Solution for IoT • Spoilt for Choice for a Digital Twin • Global IoT Architectures • A Digital Twin for 100000 Connected Cars 27
  28. 28. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de 600+ IoT Platforms 28 https://iot-analytics.com/iot-platform-companies-landscape-2020/
  29. 29. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Proprietary IoT Platforms • Sophisticated integration for related IIoT protocols (like Siemens S7, Modbus, etc.) and standards (like OPC-UA) • Not a single product (plenty of acquisitions, OEMs and different code bases are typically the foundation) • Typically very expensive • Proprietary (just open interfaces) • Limited scalability 29
  30. 30. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de IoT Offerings from Cloud Providers • Sophisticated tools for IoT management (devices, shadowing, …) • Good integration with other cloud services (storage, analytics, …) • Vendor lock-in • No focus on hybrid and edge (but on prem products) • Limited scalability • Often high cost (beyond ’hello world’) 30
  31. 31. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Standards-based / Open Source IoT Platforms • Open and standards-based (e.g. MQTT) • Open source / open core business model • Infrastructure-independent • Different vendors behind the core technologies • Sometimes less mature or non-existent connectivity (especially to legacy and proprietary protocols) • Trade-off: Solid offering for one standard (e.g. HiveMQ for MQTT) or diversity but not for mission-critical scale (e.g. Node- RED) 31
  32. 32. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • IoT Platforms • Apache Kafka as Event Streaming Solution for IoT • Spoilt for Choice for a Digital Twin • Global IoT Architectures • A Digital Twin for 100000 Connected Cars 32
  33. 33. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de The Log ConnectorsConnectors Producer Consumer Streaming Engine Apache Kafka - The Rise of an Event Streaming Platform 33 = Messaging + Storage + Integration + Processing
  34. 34. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Apache Kafka at Scale at Tech Giants > 7 trillion messages / day > 6 Petabytes / day “You name it” * Kafka is not just used for big data ** Kafka Is not just used by tech giants 34
  35. 35. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in- car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $↔ Example Case Studies (of many) Confluent - Business Value per Use Case 35
  36. 36. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de P Decoupling of Producers and Consumers Time C2 C3C1 36
  37. 37. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Schema Enforcement (Client and Server Side) 37
  38. 38. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Kafka Connect Kafka Cluster MQTT Integration Domain-Driven Design (DDD) for your Event Steaming Platform OPC-UA Integration Real Time Predictions IoT Platform Connector Java / Python / ”you-name-it” Schema Registry Event Streaming Platform IoT Domain Legacy Domain Analytics Domain è Independent and loosely coupled, but scalable, highly available and reliable! 38
  39. 39. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de 10 Reasons for Event Streaming with Apache Kafka Real Time Scalable Cost Reduction 24/7 – Zero downtime, zero data loss Decoupling – Storage, Domain-driven Design Data (re-)processing and stateful client applications Integration – Connectivity to IoT, legacy, big data, everything Hybrid Architecture – On Premises, multi cloud, edge computing Fully managed cloud No vendor locking 39
  40. 40. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Twin and AI / Machine Learning (with Kafka) • Complementary Concepts • Continuous Learning, Monitoring and Acting à Real time, scalable • (Good) data is key for success à Integration, data processing 40
  41. 41. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Apache Kafka and Machine Learning – More Details 41 Blog: How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka: https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/ Blog: Using Apache Kafka to Drive Cutting-Edge Machine Learning: https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning/ Blog: Machine Learning and Real-Time Analytics in Apache Kafka Applications: https://www.confluent.io/blog/machine-learning-real-time-analytics-models-in-kafka-applications/ Blog: Streaming Machine Learning with Tiered Storage and Without a Data Lake: https://www.confluent.io/blog/streaming-machine-learning-with-tiered-storage/ Slides + Video: Event-driven Model Scoring vs. RPC with Model Server: https://www.confluent.io/kafka-summit-san-francisco-2019/event-driven-model-serving-stream-processing-vs-rpc-with-kafka- and-tensorflow/ Few examples for real time scoring with Kafka Steams and TensorFlow, H2O, etc.: https://github.com/kaiwaehner/kafka-streams-machine-learning-examples Demo: Anomaly Detection with KSQL: https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot Demo + Video: Connected Car demo with Kafka + Streaming ML https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
  42. 42. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Hold on… Kafka is NOT an IoT Platform!
  43. 43. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Device management Unreliable networks Connectivity beyond standards Edge hardware …
  44. 44. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de (De facto) Standards for Processing IoT Data A Match Made In Heaven + = IoT Platform
  45. 45. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • IoT Platforms • Apache Kafka as Event Streaming Solution for IoT • Spoilt for Choice for a Digital Twin • Global IoT Architectures • A Digital Twin for 100000 Connected Cars 45
  46. 46. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Characteristics of Digital Twin Technology • Connectivity • Physical assets, enterprise software, customers • Bidirectional communication to ingest, command and control • Homogenization • Decoupling and standardization • Virtualization of information • Shared with multiple agents, unconstrained by physical location or time • Lower cost and easier testing, development and predictions • Reprogrammable and smart • Adjust and improve characteristics and develop new version of a product • Digital traces • Go back in time and analyse historical events to diagnose problems • Modularity • Design and customization of products and production modules • Tweak modules of models and machines 46
  47. 47. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Twin Mapped to Apache Kafka • Connectivity – Kafka Connect provides connectivity as scale in real time to IoT interfaces, big data solutions and cloud services. The Kafka ecosystem is complementary, NOT competitive to other Middleware and IoT Platforms. • Homogenization – Real decoupling between clients (i.e. producers and consumers) is one of the key strengths of Kafka. Schema management and enforcement leveraging different technologies (JSON Schema, Avro, Profobuf, etc.) enables data awareness and standardization. • Reprogrammable and smart – Kafka is the de facto standard for microservices for exactly this reason: Separation of concerns and domain-driven design (DDD). Deploy new decoupled applications and versions, do A/B testing, canarying. • Digital traces – Kakfa is a distributed commit log. Events are appended, stored as long as you want (potentially forever with rentention time = -1) and immutable. Seriously, what other technology could be used better to build a digital trace for a digital twin? • Modularity – The Kafka infrastructure itself is modular and scalable. This includes components like Kafka brokers, Connect, Schema Registry, REST Proxy and client applications in different languages like Java, Scala, Python, Go, .NET, C++ and others. With this modularity, you can easily build the right Digital Twin architecture your your edge, hybrid or global scenarios and also combine the Kafka components with any other IoT solutions. 47
  48. 48. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Simplified Pipeline for a Digital Twin 48 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS IoT Platform Digital Twin Real Time App Batch App Request Response App Connectivity Homogenization Reprogrammable and smart Digital traces Modularity
  49. 49. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Scenario 1: Digital Twin Monolith 49 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS IoT Platform Digital Twin Device Mgt. Analytics Connectivity Homogenization Reprogrammable and smart Digital traces Modularity
  50. 50. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Scenario 2: Digital Twin as External Database 50 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS IoT Platform Digital Twin Device Mgt. Database XYZAnalytics Connectivity Homogenization Reprogrammable and smart Digital traces Modularity
  51. 51. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Apache Kafka Scenario 3: Kafka as Backbone for the Digital Twin and the Rest of the Enterprise 51 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS IoT Platform Digital Twin Database XYZ Real Time App Batch App Request Response App Kafka Connect Connectivity Homogenization Reprogrammable and smart Digital traces Modularity
  52. 52. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Apache Kafka Scenario 4: Kafka as IoT Platform 52 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS IoT Cloud Platform Digital Twin Real Time App Batch App Request Response App Kafka Connect Connectivity Homogenization Reprogrammable and smart Digital traces Modularity Storage Processing
  53. 53. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Apache Kafka Scenario 5: Kafka as Digital Twin 53 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS Digital Twin Real Time App Batch App Request Response App Kafka Connect Storage Processing Connectivity Homogenization Reprogrammable and smart Digital traces Modularity
  54. 54. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Kafka as Database? Seriously?
  55. 55. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de P Kafka’s Storage – A Distributed Commit Log Time C2 C3C1 55
  56. 56. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Tiered Storage for Kafka Object Store Processing Storage Transactions, auth, quota enforcement, compaction, ... Local Remote Kafka Apps (Only available in Confluent Platform) www.kai-waehner.de | @KaiWaehner
  57. 57. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Distributed System with Replication and High Availability on Server and Client Side read, write Kafka Client Kafka Server Side Materialized View in the Client App (In-memory, RocksDB)
  58. 58. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Stateful Kafka Client Applications 58
  59. 59. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Kafka as Data Storage? 59 https://www.kai-waehner.de/blog/2020/03/12/can-apache-kafka-replace-database-acid-storage-transactions-sql-nosql-data-lake/
  60. 60. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • IoT Platforms • Apache Kafka as Event Streaming Solution for IoT • Spoilt for Choice for a Digital Twin • Global IoT Architectures • A Digital Twin for 100000 Connected Cars 60
  61. 61. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de No matter which Digital Twin Architecture I use… Most Architectures are Hybrid (Edge, Data Center, Cloud) and sometimes even Global!
  62. 62. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Edge Digital Twin Single Kafka Broker (or Cluster) Digital Twin Self-managed or certified OEM Hardware Kafka Cluster in DC / Cloud Replicator Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
  63. 63. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Centralized Digital Twin Digital Twin Single Kafka Broker (or Cluster) Self-managed or certified OEM Hardware Siemens S7, Modbus, Allen Bradley, Beckhoff ADS Single Kafka Broker (or Cluster) Self-managed or certified OEM Hardware Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
  64. 64. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Global Digital Twin Architecture Multiple Clusters and Aggregation Factories à Analytics Cluster Multi-Region Cluster High Availability (Disaster Recovery) Global Data Streaming Outsourced Development
  65. 65. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Example of a Multi-Region Digital Twin Deployment Order Sensor Order Sensor Logs Logs synchronous asynchronous ● Automate Disaster Recovery ● Sync or Async Replication per Topic ● Offset Preserving ● Automated Client Failover with No Custom Code Zero downtime, zero data loss (even in cases of data center outage) US West US East
  66. 66. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments www.kai-waehner.de | @KaiWaehner https://www.kai-waehner.de/blog/2020/01/29/deployment-patterns-distributed-hybrid-edge-global-multi-data-center-kafka-architecture/
  67. 67. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • IoT Platforms • Apache Kafka as Event Streaming Solution for IoT • Spoilt for Choice for a Digital Twin • Global IoT Architectures • A Digital Twin for 100000 Connected Cars 67
  68. 68. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de A Digital Twin with Kafka and TensorFlow 68 MQTT Proxy Elastic Grafana Kafka Cluster Kafka Connect Car Sensors Kafka Ecosystem TensorFlow Other Components Kafka Streams (Java) All Data Critical Data Ingest Data Potential Detect KSQL TensorFlow Train Analytic Model Consume Data Preprocess Data Analytic Model Deploy Analytic Model Python https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference Connectivity Homogenization Reprogrammable and smart Digital traces Modularity
  69. 69. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Architecture for 100000 Connected Cars Kafka + KSQL + MQTT + TensorFlow + Kubernetes 69 https://www.kai-waehner.de/blog/2019/11/08/live-demo-iot-100-000-connected-cars-kubernetes-kafka-mqtt-tensorflow/
  70. 70. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de Key Takeaways • A Digital Twin merges the physical and the digital world • Apache Kafka enables an open, scalable and reliable infrastructure for a Digital Twin • Event Streaming complements IoT platforms and other backend applications / databases. + 70
  71. 71. Kai Waehner Technology Evangelist contact@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Let’s connect!

×