Top 5 Event Streaming Use Cases for 2021 with Apache Kafka

1. The Top 5 Event Streaming Use Cases & Architectures in 2021 Hybrid Architectures, Edge Computing, Machine Learning, Cybersecurity, Service Mesh Kai Waehner Field CTO contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de

2. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Gartner Top Strategic Technology Trends for 2021 https://www.gartner.com/smarterwithgartner/gartner-top-strategic-technology-trends-for-2021/

3. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC Event Streaming

4. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Global Scale Real-time Persistent Storage Stream Processing Data Integration Apache Kafka The De-facto Standard for Real-Time Event Streaming Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka

5. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $↔ Example Case Studies (of many)

6. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity

8. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Why Kafka in Multiple Data Centers? * Not a representative survey J ** ‘Many DCs’ does NOT necessarily mean more than one Kafka Cluster

9. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Disaster Recovery – RPO and RTO RPO = Recovery Point Objective RTO = Recovery Time Objective

10. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Disaster Recovery @ JPMorgan https://www.confluent.io/kafka-summit-san-francisco-2019/secure-kafka-at-scale-in-true-multi-tenant-environment

11. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cluster Linking • Hybrid-cloud and multi-cloud • No additional infrastructure (such as Kafka Connect or MirrorMaker) • Just configuration • Regional or global

12. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent What is the right Hybrid Kafka Architecture for you? (Hint: This is hard à Let’s guide you by our experts) 12 Latency > 50ms Latency < 50ms RTO = 0 RTO > 0 RPO = 0 RPO > 0 Single Region Multi-Region Global Stretched Cluster x x x x Replicator x x x x* x Cluster-Linking x x x x* x MRC Sync x x x x** MRC Observer x x x x** * With a stretched cluster in a single region, you still have RTO & RPO = 0 ** Requires 3 regions minimum

13. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Kafka as a Service – Fully Managed? Infrastructure management (commodity) Scaling ● Upgrades (latest stable version of Kafka) ● Patching ● Maintenance ● Sizing (retention, latency, throughput, storage, etc.) ● Data balancing for optimal performance ● Performance tuning for real-time and latency requirements ● Fixing Kafka bugs ● Uptime monitoring and proactive remediation of issues ● Recovery support from data corruption ● Scaling the cluster as needed ● Data balancing the cluster as nodes are added ● Support for any Kafka issue with less than X minutes response time Infra-as-a-Service Harness full power of Kafka Kafka-specific management Platform-as-a-Service Evolve as you need Future-proof Mission-critical reliability Most Kafka-as-a-Service offerings are partially-managed Kafka as a Service should be a serverless experience with consumption-based pricing!

15. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent What is the “Edge” for Kafka? • Edge is NOT a data center • Kafka clients AND the Kafka broker(s) • Offline business continuity • Often 100+ locations • Low-footprint and low-touch • Hybrid integration Example: Single broker, 1 GB Ram 100 MB/sec

16. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent CRM 3rd party payment provider Context-specific real-time upsell Customer data Payment processing and fraud detection as a service Manager Get report API Customer Customer Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Hybrid Architecture

17. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Food Inventory Loyalty System Traveler Information Orders Upsell to first class Customer data Train schedule Payment data Loyalty information Streams of real time events Updated SchedulesEvent Streaming at the Edge

18. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Data Processing at the Edge Time P C1 C2 C3 Know-your-customer Loyalty app, predictive behavior, … Estimated time of arrival Connect to the gaming server for kids Play games, earn rewards, communicate with other kids in the train, … Always on (even “offline”) Replayability Reduced traffic cost Better latency

19. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Edge Kafka @ Royal Caribbean https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/

20. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Edge Integration and Analytics @ WPX Energy Edge processing and replication to the cloud in real-time at scale in the oil&gas industry

22. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Microservices to the rescue? • Significant Operations Overhead • Substantial DevOps Skills Required • Implicit Interfaces • Duplication Of Effort http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html • Distributed System Complexity • Asynchronicity Is Difficult • Testability Challenges 22

23. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Microservices can lead to Death-star Architectures Netflix: https://www.slideshare.net/brucewong3/the-case-for-chaos Twitter: https://twitter.com/adrianco/status/441883572618948608 Hail-o: http://www.sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-3/ 450+ microservices 500+ microservices 500+ microservices 23

24. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Key Requirements for Microservices •Decoupled •Flexible •Operationally Transparent •Data Aware •Elastic 24

25. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Kafka Connect Kafka Cluster CRM Integration Domain-Driven Design (DDD) for your Microservice Architecture Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain è Independent and loosely coupled, but scalable, highly available and reliable! 25

27. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cloud-Native Deployment leveraging Kubernetes 27

28. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Service Mesh A microservice pattern to move visibility, reliability, and security primitives for service-to-service communication into the infrastructure layer, out of the application layer. https://www.infoq.com/articles/linkerd-v2-production-adoption/ 28

29. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Service Proxy Features • Metrics without instrumenting apps • Trace flow of requests across services • One stable URI for each service • Service discovery • Monitor request latency • Routing - A/B testing, green/blue deployments • Circuit breaking • Protocol translation (HTTP, gRPC, Kafka Protocol, etc.) • Mutual TLS (mTLS) • SSL Termination • Integrate with 3rd party tools like Prometheus, Grafana, Zipkin, etc. • Much more… Observability “is by far the most important thing that a Proxy and the Service Mesh provide in a distributed Microservice architecture!” Matt Klein 29

30. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Example: Lyft today with “envoy” Proxy • 100% (!!!) communication coverage - Everything talks through Envoy Proxies • à Make monitoring, debugging, firefighting as consistent as possible https://www.youtube.com/watch?v=55yi4MMVBi4 Matt Klein at QCon NY 2018 30

31. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Kafka Connect Kafka Cluster CRM Integration Clients and Servers are Independent (including their Ops Teams) Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain Proxy Proxy Proxy Proxy Proxy Proxy Control Plane 31

32. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Example: Kafka + Confluent REST Proxy Envoy Proxy I am using REST too! Kafka? Never heard of her. I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka Confluent REST Proxy I support only TCP! HTTP HTTP 32

33. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Example: Kafka + Envoy Kafka Protocol Filter Envoy Proxy I am using REST too! Kafka? Never heard of her. I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka HTTP TCP (Kafka Protocol) 33

34. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Implementation: Kafka + Istio @ Banzai Cloud 34 https://banzaicloud.com/blog/kafka-on-istio-performance/

35. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Implementation: Kafka + Istio 35 https://banzaicloud.com/blog/kafka-on-istio-performance/

36. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Implementation: Kafka + ksqlDB + Istio 36 https://banzaicloud.com/blog/supertubes-ksql/

37. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent (Potential) Features for Kafka + Service Mesh Implementation Protocol conversion from HTTP / gRPC to Kafka • Tap feature to dump to a Kafka stream • Protocol parsing for observability (stats, logging, and trace linking with HTTP RPCs) • Shadow requests to a Kafka stream instead of HTTP / gRPC shadow • Integrate with Kafka Connect and its whole ecosystem of connectors Validation of Events • Serialization format (JSON, Avro, Protobuf, etc.) • Message schema • Headers, attributes, etc. Security • SSL Termination • Mutual TLS (mTLS) • Authorization Proxy features • Dynamic Routing • Rate limiting at both the L4 connection and L7 message level • Filter, add compression, … • Automatic topic name conversion (e.g. for canary release or blue/green deployment) Monitoring and Tracing • Request logs and stats • Data lineage / audit log • Audit log by taking request logs and enriching them with the user info. • Client specific metrics (Byte rate per client id / per consumer groups, versions of the client libraries, consumer lag monitoring for the entire data center) 37

40. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Apache Kafka as Infrastructure for ML

41. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Apache Kafka’s Open Ecosystem as Infrastructure for ML Kafka Streams/ ksqlDB Kafka Connect Confluent REST Proxy Confluent Schema Registry Go/.NET/Python Kafka Producer ksqlDB Python Client

42. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Streaming Analytics for Predictive Maintenance at Scale 42 IoT Integration Layer Batch Analytics Platform BI Dashboard Streaming Platform Big Data Integration Layer Car Sensors Streaming Platform Other Components Real Time Monitoring System All Data Critical Data Ingest Data Potential Detect

43. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Streaming Analytics for Predictive Maintenance at Scale 43 IoT Integration Layer Batch Analytics Platform BI Dashboard Streaming Platform Big Data Integration Layer Car Sensors Streaming Platform Analytics Platform Other Components Real Time Monitoring System All Data Critical Data Ingest Data Potential DetectAnalytics Platform Train Analytic Model Data Processing Analytic Model Preprocess Data Consume Data Deploy Analytic Model

44. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Preprocessing with ksqlDB 44 SELECT car_id, event_id, car_model_id, sensor_input FROM car_sensor c LEFT JOIN car_models m ON c.car_model_id = m.car_model_id WHERE m.car_model_type ='Audi_A8';

45. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Direct streaming ingestion for model training with TensorFlow I/O + Kafka Plugin (no additional data storage like S3 or HDFS required!) Time Model BModel A Producer Distributed Commit Log Streaming Ingestion and Model Training with TensorFlow IO https://github.com/tensorflow/io 45 Model X (at a later time)

46. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Confluent Tiered Storage for Kafka 46 (Only available in Confluent Platform)

47. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Local Predictions Model Training in the Cloud Model Deployment at the Edge Analytic Model Separation of Model Training and Model Inference 47

48. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent “CREATE STREAM AnomalyDetection AS SELECT sensor_id, detectAnomaly(sensor_values) FROM car_engine;“ User Defined Function (UDF) Model Deployment with Apache Kafka, ksqlDB and TensorFlow 48

50. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cybersecurity The threat is real! Challenges Stealing IP DDoS Ransomware / wiperware WannaCry, NotPetya, … Damage: Billions of dollars ”Supply chain attack” Digital Transformation Networking Communication Connectivity Open standards ”Always-on”

51. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Legacy SIEM needs to evolve ForwarderNetwork traffic Firewall logs RDBMS Application logs Adaptors Beats Sensor Data Challenges: ● Proprietary forwarders that can only send data to single source ● Data locked from being shared ● Difficult to scale with growing data volumes ● Prohibitively high indexing costs ● Unable to filter out noisy data ● Slow batch processing HTTP proxy logs

52. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent AI/ML Modernized security information and event management (SIEM) Filter, transform aggregate APP SIEM Index Search Curated streams Forensic Archive HDFS S3 Big Query CDC Syslog Network traffic Firewall logs RDBMS Application logs Payment Data HTTP proxy logs QRadar Arcsight Splunk Elastic APP Stateful real-time analytics

53. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cyber Intelligence Platform leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more… https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html

54. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Fraud Detection at Scale in Real-Time for Billions of Messages https://www.infoq.com/presentations/paypal-data-service-fraud https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html

55. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent How does Confluent help?

56. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Rise of Event Streaming 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 Companies trust and use Apache Kafka

57. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi- Region) Build Contextual Event- Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management...

58. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Confluent Platform Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional Services Enterprise Support Apache Kafka EFFICIENT OPERATIONS AT SCALE PRODUCTION- STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY SQL-based Stream Processing KSQL (ksqlDB) Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development non-Java clients | REST Proxy GUI-driven Mgmt & Monitoring Control Center Flexible DevOps Automation Operator | Ansible Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Enterprise-grade Security RBAC | Secrets | Audit logs Data Compatibility Schema Registry | Schema Validation Global Resilience Multi-Region Clusters | Replicator Developer Operator Architect Open Source | Community licensed PARTNERSHIP FOR BUSINESS SUCCESS Complete Engagement Model Revenue / Cost / Risk Impact TCO / ROI Executive Buyer

59. Kai Waehner Field CTO contact@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Let’s connect!

Top 5 Event Streaming Use Cases for 2021 with Apache Kafka

Kai Wähner

Top 5 Event Streaming Use Cases for 2021 with Apache Kafka