Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to JHipster conf 2019 - Kafka Ecosystem
Similar to JHipster conf 2019 - Kafka Ecosystem (20)
Recently uploaded
Recently uploaded (20)
JHipster conf 2019 - Kafka Ecosystem
- 1. 1 1 Apache Kafka and Jhipster ! Florent Ramière TAM / System Engineer florent@confluent.io
- 2. 2 Massive volumes of new data generated every day Mobile Cloud Microservices Internet of Things Machine Learning Distributed across apps, devices, datacenters, clouds Structured, unstructured polymorphic What
- 3. 3 Problem ?
- 4. 4 Silos explained by Data Gravity concept As data accumulates (builds mass) there is a greater likelihood that additional services and applications will be attracted to this data. This is the same effect gravity has on objects around a planet. As the mass or density increases, so does the strength of gravitational pull.
- 5. 5 With
- 6. 6 How
- 7. 7 Store & ETL Process Publish & Subscribe In short
- 8. 8 From a simple idea
- 9. 9 From a simple idea
- 10. 10 with great properties ! • Scalability • Retention • Durability • Replication • Security • Resiliency • Throughput • Ordering • Exactly Once Semantic • Transaction • Idempotency • Immutability • …
- 11. 11 11 Producer
- 12. 12 Anatomy of a Message
- 13. 13
- 15. 15 Set up secure Kafka & build your first app Understand streaming Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Infrastructure & apps across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Self-service on shared Kafka Infrastructure & applications across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streamingUnderstand streaming Pre-streamingValue Stream Everything 05Break Silos 04 03 Go To Production 02 Learn Kafka 01 Investment & Time Solve A Critical Need Maturity model
- 16. 16 Set up secure Kafka & build your first app Understand streaming Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Infrastructure & apps across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Self-service on shared Kafka Infrastructure & applications across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streamingUnderstand streaming Pre-streamingValue Stream Everything 05Break Silos 04 03 Go To Production 02 Learn Kafka 01 Solve A Critical Need Maturity model
- 17. 17 Set up secure Kafka & build your first app Understand streaming Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Infrastructure & apps across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Self-service on shared Kafka Infrastructure & applications across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streamingUnderstand streaming Pre-streamingValue Stream Everything 05Break Silos 04 03 Go To Production 02 Learn Kafka 01 Solve A Critical Need Maturity model
- 19. 19
- 21. 21 21 This is a full platform
- 22. 22 … spawned a full platform Apache Kafka® Core | Connect API | Streams API Stream Processing & Compatibility KSQL | Schema Registry Operations Replicator | Auto Data Balancer | Connectors | MQTT Proxy | Operator Database Changes Log Events IoT Data Web Events other events Hadoop Database Data Warehouse CRM other DATA INTEGRATION Transformations Custom Apps Analytics Monitoring other REAL-TIME APPLICATIONS OPEN SOURCE FEATURES COMMERCIAL FEATURES Datacenter Public Cloud Confluent Cloud CONFLUENT PLATFORM Administration & Monitoring Control Center | Security Connectivity Clients | Connectors | REST Proxy CONFLUENT FULLY-MANAGEDCUSTOMER SELF-MANAGED
- 23. 23 23 ETL
- 24. 24 I
- 26. 26
- 27. 27
- 28. 28
- 29. 29
- 30. 30
- 31. 31
- 33. 33
- 34. 34 T1
- 35. 35 T1,T2,T3
- 43. 43 Apache Kafka Connect API: Import and Export Data In & Out of Kafka JDBC Mongo MySQL Elastic Cassandra HDFS Kafka Connect API Kafka Pipeline Connector Connector Connector Connector Connector Connector Sources Sinks Fault tolerant Manage hundreds of data sources and sinks Preserves data schema Integrated within Confluent Control Center
- 44. 44 Connectors: Connect Kafka Easily with Data Sources and Sinks Databases Datastore/File Store Analytics Applications / Other
- 45. 45 Kafka Connect API, Part of the Apache Kafka™ Project Connect any source to any target system Integrated • 100% compatible with Kafka v0.9 and higher • Integrated with Confluent’s Schema Registry • Easy to manage with Confluent Control Center Flexible • 40+ open source connectors available • Easy to develop additional connectors • Flexible support for data types and formats Compatible • Maintains critical metadata • Preserves schema information • Supports schema evolution Reliable • Automated failover • Exactly-once guarantees • Balances workload between nodes
- 46. 46 Confluent Hub - The Kafka App Store
- 48. 48 Clients: Communicate with Kafka in a Broad Variety of Languages Apache Kafka Confluent Platform Community Supported Proxy http/REST stdin/stdout Confluent Platform Clients developed and fully supported by Confluent
- 49. 49 REST Proxy: Talking to Non-native Kafka Apps and Outside the Firewall REST Proxy Non-Java Applications Native Kafka Java Applications Schema Registry REST / HTTP Simplifies administrative actions Simplifies message creation and consumption Provides a RESTful interface to a Kafka cluster
- 50. 50 50 Processing
- 51. 51 Stream Processing by Analogy Kafka Cluster Connect API Stream Processing Connect API $ cat < in.txt | grep "ksql" | tr a-z A-Z > out.txt
- 52. 52 • subscribe() • poll() • send() • flush() Consumer, Producer Flexibility Simplicity Trade offs
- 54. 54 JHipster - Spring helping Consumer Producer
- 55. 55 • subscribe() • poll() • send() • flush() Consumer, Producer • mapValues() • filter() • punctuate() Kafka Streams Flexibility Simplicity Trade offs
- 58. Consumer Group Protocol Power! App Streams API App Streams API App Streams API Same app, many instances
- 61. 61 Things Kafka Streams Does Runs everywhere Clustering done for you Exactly-once processing Event-time processing Integrated database Joins, windowing, aggregation S/M/L/XL/XXL/XXXL sizes
- 62. 62 • subscribe() • poll() • send() • flush() Consumer, Producer • mapValues() • filter() • punctuate() Kafka Streams Flexibility Simplicity Trade offs
- 63. 63 63 Kafka Streams Time time time
- 64. 64 Time! Time! Time! Time! Time! Time! Time! Time!
- 65. 65 Windowing in Kafka Streams
- 69. 69 Event Time Processing Event-time ”The point in time when an event or data record occurred, i.e. was originally created "by the source". Achieving event-time semantics typically requires embedding timestamps in the data records at the time a data record is being produced.” Processing-time ”The point in time when the event or data record happens to be processed by the stream processing application, i.e. when the record is being consumed. The processing-time may be milliseconds, hours, or days etc. later than the original event- time.” Ingestion-time “The point in time when an event or data record is stored in a topic partition by a Kafka broker.”
- 70. 70 70 Kafka Streams Exactly once semantic
- 71. 71 Delivery Guarantee At most once “Messages may be lost but are never redelivered.” At least once “Messages are never lost but may be redelivered.“ Exactly once “Each message is delivered once and only once.“
- 73. 73 • subscribe() • poll() • send() • flush() Consumer, Producer • mapValues() • filter() • punctuate() Kafka Streams • Select…from… • Join…where… • Group by.. KSQL Flexibility Simplicity Trade offs
- 74. 74 KSQL for Data Exploration SELECT status, bytes FROM clickstream WHERE user_agent = 'Mozilla/5.0 (compatible; MSIE 6.0)';
- 75. 75 KSQL for Streaming ETL Fact 1 Fact 2 Fact 3 Fact 4 Fact 5 Fact 6 id 1 id 2 id 3Business
- 76. 76 KSQL for Streaming ETL Fact 1 Fact 2 Fact 3 Fact 4 Fact 5 Fact 6 id 1 id 2 id 3 Fact X Fact Y Fact Z
- 77. 77 KSQL for Streaming ETL Fact 1 Fact 2 Fact 3 Fact 4 Fact 5 Fact 6 id 1 id 2 id 3 Fact X Fact Y Fact Z Fact A Fact B Fact C Fact D Fact E Fact K Fact L Fact M Fact N Id X
- 78. 78 KSQL for Streaming ETL CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum';
- 79. 79 Nested Types SELECT eventid, address.city FROM users WHERE address.state = 'CA';
- 80. 80 User Defined Functions (UDF) SELECT eventid, anomaly(sensorinput) FROM sensor @Udf(description = "apply analytic model to sensor input") public String anomaly(String sensorinput){ return your_logic; }
- 81. 81 KSQL for Anomaly Detection CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 SECONDS) GROUP BY card_number HAVING count(*) > 3;
- 82. 82
- 83. 83 Plenty of KSQL Recipies https://www.confluent.io/stream-processing-cookbook/
- 84. 84 Plenty of KSQL Recipies https://www.confluent.io/stream-processing-cookbook/
- 85. 85 Plenty of KSQL Recipies https://www.confluent.io/stream-processing-cookbook/
- 86. 86 KSQL: Enable Stream Processing using SQL-like Semantics Example Use Cases • Streaming ETL • Anomaly detection • Event monitoring Leverage Kafka Streams API without any coding required KSQL server Engine (runs queries) REST API CLIClients Confluent Control Center GUI Kafka Cluster Use any programming language Connect via CLI or Control Center user interface
- 87. 87 KSQL is really Kafka Stream ? ... yes!
- 88. 88 • subscribe() • poll() • send() • flush() Consumer, Producer • mapValues() • filter() • punctuate() Kafka Streams • Select…from… • Join…where… • Group by.. KSQL Flexibility Simplicity Trade offs
- 89. 89 Lowering the Bar to Enter the World of Streaming Kafka User Population CodingSophistication Core Java developers Core developers who don’t use Java/Scala Data engineers, architects, DevOps/SRE BI analysts streams
- 90. 90 90 Schema
- 91. 91 The Challenge of Data Compatibility at Scale : implicit à explicit ! App 1 App 2 App 3 Many sources without a policy causes mayhem in a centralized data pipeline Ensuring downstream systems can use the data is key to an operational stream pipeline Example: Date formats Even within a single application, different formats can be presented Incompatibly formatted message
- 92. 92 Schema Registry: Make Data Backwards Compatible and Future-Proof ● Define the expected fields for each Kafka topic ● Automatically handle schema changes (e.g. new fields) ● Prevent backwards incompatible changes ● Support multi-data center environments Elastic Cassandra HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic! Schema Registry
- 93. 93 93 Resources
- 95. 95 Optimizing Your Apache Kafka® Deployment https://www.confluent.io/white-paper/optimizing-your-apache-kafka-deployment/
- 96. 96 Resources - Confluent Enterprise Reference Architecture https://www.confluent.io/whitepaper/confluent-enterprise-reference-architecture/
- 97. 97 97 Community
- 98. 98 Resources – Community Slack and Mailing List https://slackpass.io/confluentcommunity https://groups.google.com/forum/#!forum/confluent-platform
- 100. 100 Confluent Platform Demo : cp-demo https://github.com/confluentinc/cp-demo With security inside!
- 101. 101 Examples Examples Examples ! https://github.com/confluentinc/examples
- 104. 104 104 Take Away
- 105. 105 JHipster provides a quick start application for the Modern Digital Enterprise Enabling developers of the whole to focus on business
- 106. 106 Kafka Provides a Central Nervous System for the Modern Digital Enterprise Enabling companies to respond accurately and in real time to business events
- 107. 107 JHipster and Kafka Provide a great set of combinations of properties for the Modern Digital Enterprise Enabling developers to move faster to help companies respond in real time to business events.