Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
gschmutz
Solutions for bi-directional integration
between Oracle RDBMS & Apache Kafka
Analytics & Data Summit 2019
Guido S...
gschmutz
Agenda
Bi-directional integration between Oracle RDBMS & Apache Kafka
1. Introduction
2. Blueprints Oracle RDBMS ...
gschmutz
Guido Schmutz
Bi-directional integration between Oracle RDBMS & Apache Kafka
Working at Trivadis for more than 22...
gschmutzBi-directional integration between Oracle RDBMS & Apache Kafka
Introduction
gschmutz
Timely decisions require new data in minutes
Bi-directional integration between Oracle RDBMS & Apache Kafka
gschmutz
Hadoop Clusterd
Hadoop Cluster
Big Data
Unified Architecture for Modern Data Analytics Solutions
Bi-directional i...
gschmutz
Hadoop Clusterd
Hadoop Cluster
Big Data
Unified Architecture for Modern Data Analytics Solutions
Bi-directional i...
gschmutz
Highly available, Pub/Sub infrastructure
Highly Scalable
Event Hub: Apache Kafka
Distributed Log at the Core
Logs...
gschmutz
Use Case
Customer Microservice
{
}
Customer API CustomerCustomer Logic
Order Processing System
{
}
Order API Orde...
gschmutz
Blueprints Oracle RDBMS =>
Apache Kafka
Bi-directional integration between Oracle RDBMS & Apache Kafka
gschmutz
Blueprints Apache Kafka => Oracle RDBMS (K-DB)
Customer Microservice
{
}
Customer API CustomerCustomer Logic
Orde...
gschmutz
DB-K_1: Regular Polling of RDBMS table/view
Bi-directional integration between Oracle RDBMS & Apache Kafka
Event
...
gschmutz
DB-K_1: Kafka Connect with JDBC Polling
Bi-directional integration between Oracle RDBMS & Apache Kafka
Event
Hub
...
gschmutz
Stream Data Integration: Kafka Connect
Bi-directional integration between Oracle RDBMS & Apache Kafka
• Single Me...
gschmutz
DB-K_1 – Configure JDBC Connector
Bi-directional integration between Oracle RDBMS & Apache Kafka
#!/bin/bash
curl...
gschmutz
DB-K_2: Regular Polling of RDBMS API
Bi-directional integration between Oracle RDBMS & Apache Kafka
Event
Hub
Str...
gschmutz
DB-K_2: Kafka Connect & Oracle Rest Data Service
Bi-directional integration between Oracle RDBMS & Apache Kafka
E...
gschmutz
Oracle REST Data Services (ORDS)
Bi-directional integration between Oracle RDBMS & Apache Kafka
• makes it easy t...
gschmutz
DB-K_2 – Setup ORDS (I)
Bi-directional integration between Oracle RDBMS & Apache Kafka
ORDS.ENABLE_SCHEMA(
p_enab...
gschmutz
DB-K_2 – Setup ORDS (II)
Bi-directional integration between Oracle RDBMS & Apache Kafka
ORDS.DEFINE_HANDLER(
p_mo...
gschmutz
Stream Data Integration: StreamSets
Continuous open source, intent-driven,
big data ingest
Visible, record-orient...
gschmutz
DB-K_2 – StreamSets Data Flow
Bi-directional integration between Oracle RDBMS & Apache Kafka
gschmutz
DB-K_3: Change Data Capture (CDC) on RDBMS
Bi-directional integration between Oracle RDBMS & Apache Kafka
Stream ...
gschmutz
DB-K_3: Oracle GoldenGate and Kafka Connect / Rest
Bi-directional integration between Oracle RDBMS & Apache Kafka...
gschmutz
DB-K_3: Debezium and Kafka Connect
Bi-directional integration between Oracle RDBMS & Apache Kafka
Stream Data
Int...
gschmutz
DB-K_4: Produce Messages to Event Hub from RDBMS
Bi-directional integration between Oracle RDBMS & Apache Kafka
E...
gschmutz
DB-K_4: Produce Messages to REST Proxy
Bi-directional integration between Oracle RDBMS & Apache Kafka
Event
Hub
S...
gschmutz
DB-K_4: Oracle Big Data SQL writes to Kafka topic
Bi-directional integration between Oracle RDBMS & Apache Kafka
...
gschmutz
DB-K_5: Queuing on RDBMS to bridge to Event Hub
Bi-directional integration between Oracle RDBMS & Apache Kafka
St...
gschmutz
DB-K_5: Oracle Advanced Queuing & Kafka Connect JMS
Bi-directional integration between Oracle RDBMS & Apache Kafk...
gschmutz
BEGIN
dbms_aqadm.create_queue_table (
queue_table => 'ORDER_QT',
queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE'...
gschmutz
curl -X "POST" "$DOCKER_HOST_IP:8083/connectors" 
-H "Content-Type: application/json" 
--data '{
"name": "jms-sou...
gschmutz
DB-K_5: Oracle Advanced Queuing & MirrorMaker
Bi-directional integration between Oracle RDBMS & Apache Kafka
Stre...
gschmutzBi-directional integration between Oracle RDBMS & Apache Kafka
Blueprints Apache Kafka => Oracle
RDBMS
gschmutz
Blueprints Apache Kafka => Oracle RDBMS (K-DB)
Bi-directional integration between Oracle RDBMS & Apache Kafka
Cus...
gschmutz
K-DB_1: Write directly to RDBMS table/view
Bi-directional integration between Oracle RDBMS & Apache Kafka
Event
H...
gschmutz
K-DB_1: Write to RDBMS table/view
Bi-directional integration between Oracle RDBMS & Apache Kafka
Event
Hub
Stream...
gschmutz
K-DB_2: Write over RDBMS API
Bi-directional integration between Oracle RDBMS & Apache Kafka
Event
Hub
Stream Data...
gschmutz
K-DB_2: Write over Oracle REST Database service
Bi-directional integration between Oracle RDBMS & Apache Kafka
Ev...
gschmutz
DB-K_2 – Setup ORDS (I)
Bi-directional integration between Oracle RDBMS & Apache Kafka
ORDS.DEFINE_HANDLER(
p_mod...
gschmutz
K-DB_3: Consume from Event Hub
Bi-directional integration between Oracle RDBMS & Apache Kafka
Event
Hub
Stream Da...
gschmutz
K-DB_3: Oracle Big Data SQL exposes topic as table
Bi-directional integration between Oracle RDBMS & Apache Kafka...
gschmutz
K-DB_4: Queuing on RDBMS to bridge with Event Hub
Bi-directional integration between Oracle RDBMS & Apache Kafka
...
gschmutz
K-DB_4: Use AQ on RDBMS to bridge with Event Hub
Bi-directional integration between Oracle RDBMS & Apache Kafka
S...
gschmutz
K-DB_4: Use AQ on RDBMS to bridge with Event Hub
Bi-directional integration between Oracle RDBMS & Apache Kafka
S...
gschmutzBi-directional integration between Oracle RDBMS & Apache Kafka
Demo
gschmutz
Demo
Customer Microservice
{
}
Customer API CustomerCustomer Logic
Order Processing System
{
}
Order API OrderOrd...
gschmutzBi-directional integration between Oracle RDBMS & Apache Kafka
Technology on its own won't help you.
You need to k...
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

4

Share

Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka

Download to read offline

Apache Kafka is a popular distributed streaming data platform. A Kafka cluster stores streams of records (messages) in categories called topics. It is the architectural backbone of modern data analytics. Data flowing into Kafka often originates from native data streams such as social media streams, telemetry data, financial transactions and many others. But these data streams only contain part of the information. A lot of data necessary in stream processing is stored in traditional systems backed by relational databases. To implement new and modern, real-time solutions, an up-to-date view of that information is needed. So how do we make sure that information can flow between the RDMBS and Kafka, so that changes are available in Kafka as soon as possible in near-real-time? It this session, we present different approaches for integrating relational databases with Kafka, such as Kafka Connect, Oracle GoldenGate and bridging Kafka with Oracle Advanced Queuing (AQ).

Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka

  1. 1. gschmutz Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka Analytics & Data Summit 2019 Guido Schmutz (guido.schmutz@Trivadis.com) gschmutz guidoschmutz.wordpress.com
  2. 2. gschmutz Agenda Bi-directional integration between Oracle RDBMS & Apache Kafka 1. Introduction 2. Blueprints Oracle RDBMS => Apache Kafka 3. Blueprints Apache Kafka => Oracle RDBMS 4. Demo
  3. 3. gschmutz Guido Schmutz Bi-directional integration between Oracle RDBMS & Apache Kafka Working at Trivadis for more than 22 years Oracle GroundbreakerAmbassador& Oracle ACE Director Consultant, Trainer Software Architect forJava, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board TechnologyManager @ Trivadis More than 30 years of software developmentexperience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare:http://www.slideshare.net/gschmutz Twitter: gschmutz 145th edition
  4. 4. gschmutzBi-directional integration between Oracle RDBMS & Apache Kafka Introduction
  5. 5. gschmutz Timely decisions require new data in minutes Bi-directional integration between Oracle RDBMS & Apache Kafka
  6. 6. gschmutz Hadoop Clusterd Hadoop Cluster Big Data Unified Architecture for Modern Data Analytics Solutions Bi-directional integration between Oracle RDBMS & Apache Kafka SQL Search BI Tools Enterprise Data Warehouse Search / Explore File Import / SQL Import Event Hub Parallel Processing Storage Storage RawRefined Results Microservice State { } API Stream Processor State { } API Event Stream Event Stream Service Stream Analytics Microservices Enterprise Apps Logic { } API Edge Node Rules Event Hub Storage Bulk Source Event Source Location DB Extract File DB IoT Data Mobile Apps Social Telemetry Event Stream
  7. 7. gschmutz Hadoop Clusterd Hadoop Cluster Big Data Unified Architecture for Modern Data Analytics Solutions Bi-directional integration between Oracle RDBMS & Apache Kafka SQL Search BI Tools Enterprise Data Warehouse Search / Explore File Import / SQL Import Event Hub Parallel Processing Storage Storage RawRefined Results Microservice State { } API Stream Processor State { } API Event Stream Event Stream Service Stream Analytics Microservices Enterprise Apps Logic { } API Edge Node Rules Event Hub Storage Bulk Source Event Source Location DB Extract File DB IoT Data Mobile Apps Social Telemetry Event Stream
  8. 8. gschmutz Highly available, Pub/Sub infrastructure Highly Scalable Event Hub: Apache Kafka Distributed Log at the Core Logs do not (necessarily) forget • Never • Time (TTL) or Size-based • Log-Compacted based Bi-directional integration between Oracle RDBMS & Apache Kafka
  9. 9. gschmutz Use Case Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order Customer (compacted) Notification Microservice Notification Logic Schema Registry Bi-directional integration between Oracle RDBMS & Apache Kafka
  10. 10. gschmutz Blueprints Oracle RDBMS => Apache Kafka Bi-directional integration between Oracle RDBMS & Apache Kafka
  11. 11. gschmutz Blueprints Apache Kafka => Oracle RDBMS (K-DB) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry DB-K_1: Regular Polling ofRDBMS table/view DB-K_2: Regular Polling ofRDBMS API DB-K_3: Change Data Capture (CDC) on RDBMS DB-K_4: Produce Messages to Event Hub from RDBMS DB-K_5: Queuing on RDBMS to bridge to Event Hub DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5 Bi-directional integration between Oracle RDBMS & Apache Kafka
  12. 12. gschmutz DB-K_1: Regular Polling of RDBMS table/view Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  13. 13. gschmutz DB-K_1: Kafka Connect with JDBC Polling Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  14. 14. gschmutz Stream Data Integration: Kafka Connect Bi-directional integration between Oracle RDBMS & Apache Kafka • Single Message Transforms (SMT) allow simple transformations • Connectors available from Confluent as well as the community, check hub.confluent.io for the list curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "mqtt-source", "config": { "connector.class": ”...MqttSourceConnector", "tasks.max": "1", "name": "mqtt-source", "mqtt.server.uri": "tcp://mosquitto:1883", "mqtt.topics": "truck/+/position", "kafka.topic":"truck_position", } }' • declarative style data flows • simplicity - “simple things done simple” • very well integrated with Kafka – Kafka Connect Framework is part of Apache Kafka
  15. 15. gschmutz DB-K_1 – Configure JDBC Connector Bi-directional integration between Oracle RDBMS & Apache Kafka #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "jdbc-driver-source", "config": { "connector.class": "JdbcSourceConnector", "connection.url":"jdbc:oracle:thin//oracle-db:1521/XEPDB1", "mode": "timestamp", "timestamp.column.name":”modified_at", "table.whitelist":”order", "validate.non.null":"false", "topic.prefix":”orderprocessing_", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "name": "jdbc-driver-source", "transforms":"createKey,extractInt", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"id", "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractInt.field":"id" } }'
  16. 16. gschmutz DB-K_2: Regular Polling of RDBMS API Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  17. 17. gschmutz DB-K_2: Kafka Connect & Oracle Rest Data Service Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  18. 18. gschmutz Oracle REST Data Services (ORDS) Bi-directional integration between Oracle RDBMS & Apache Kafka • makes it easy to develop modern REST interfaces for relational data in the Oracle Database and the Oracle Database 18c JSON Document Store • ORDS maps HTTP(S) verbs (GET, POST, PUT, DELETE, etc.) to database transactions and returns any results formatted using JSON • Java middle tier application on WebLogic, Tomcat, Docker, Standalone (for development)
  19. 19. gschmutz DB-K_2 – Setup ORDS (I) Bi-directional integration between Oracle RDBMS & Apache Kafka ORDS.ENABLE_SCHEMA( p_enabled => TRUE, p_schema => 'ORDER_PROCESSING', p_url_mapping_type => 'BASE_PATH', p_url_mapping_pattern => 'order_processing', p_auto_rest_auth => FALSE); ORDS.DEFINE_MODULE( p_module_name => 'order_processing', p_base_path => '/orders/', p_items_per_page => 25, p_status => 'PUBLISHED', p_comments => NULL); ORDS.DEFINE_TEMPLATE( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_priority => 0, p_etag_type => 'HASH', p_etag_query => NULL, p_comments => NULL);
  20. 20. gschmutz DB-K_2 – Setup ORDS (II) Bi-directional integration between Oracle RDBMS & Apache Kafka ORDS.DEFINE_HANDLER( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_method => 'GET', p_source_type => 'resource/lob', p_items_per_page => 25, p_comments => NULL, p_source => 'SELECT ''application/json'', json_object(''orderId'' VALUE po.id, ''orderDate'' VALUE po.order_date, ''orderMode'' VALUE po.order_mode, ''customer'' VALUE json_object(''firstName'' VALUE cu.first_name, ''lastName'' VALUE cu.last_name), ''lineItems'' VALUE (SELECT json_arrayagg( json_object(''ItemNumber'' VALUE li.id, ''Product'' VALUE json_object(''productId'' VALUE li.product_id, ''unitPrice'' VALUE li.unit_price), ''quantity'' VALUE li.quantity)) FROM order_item_t li WHERE po.id = li.order_id), ''offset'' VALUE TO_CHAR(po.modified_at, ''YYYYMMDDHH24MISS'')) FROM order_t po LEFT JOIN customer_t cu ON (po.customer_id = cu.id) WHERE po.modified_at > TO_DATE(:offset, ''YYYYMMDDHH24MISS'')'
  21. 21. gschmutz Stream Data Integration: StreamSets Continuous open source, intent-driven, big data ingest Visible, record-oriented approach fixes combinatorial explosion Both stream and batch processing • Standalone, Spark cluster, MapReduce cluster IDE for pipeline development by ‘civilians’ special option for Edge computing custom sources, sinks, processors Supported by StreamSets
  22. 22. gschmutz DB-K_2 – StreamSets Data Flow Bi-directional integration between Oracle RDBMS & Apache Kafka
  23. 23. gschmutz DB-K_3: Change Data Capture (CDC) on RDBMS Bi-directional integration between Oracle RDBMS & Apache Kafka Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  24. 24. gschmutz DB-K_3: Oracle GoldenGate and Kafka Connect / Rest Bi-directional integration between Oracle RDBMS & Apache Kafka Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Rest Proxy Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  25. 25. gschmutz DB-K_3: Debezium and Kafka Connect Bi-directional integration between Oracle RDBMS & Apache Kafka Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Alternatives: StreamSets Data Collector Attunity … Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  26. 26. gschmutz DB-K_4: Produce Messages to Event Hub from RDBMS Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  27. 27. gschmutz DB-K_4: Produce Messages to REST Proxy Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Rest Proxy ? Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  28. 28. gschmutz DB-K_4: Oracle Big Data SQL writes to Kafka topic Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  29. 29. gschmutz DB-K_5: Queuing on RDBMS to bridge to Event Hub Bi-directional integration between Oracle RDBMS & Apache Kafka Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  30. 30. gschmutz DB-K_5: Oracle Advanced Queuing & Kafka Connect JMS Bi-directional integration between Oracle RDBMS & Apache Kafka Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  31. 31. gschmutz BEGIN dbms_aqadm.create_queue_table ( queue_table => 'ORDER_QT', queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE', sort_list => 'PRIORITY,ENQ_TIME', multiple_consumers => FALSE, message_grouping => dbms_aqadm.none ); END; / DB-K_5 – Setup AQ and Kafka Connect (I) Bi-directional integration between Oracle RDBMS & Apache Kafka BEGIN dbms_aqadm.create_queue ( queue_name => 'ORDER_AQ', queue_table => 'ORDER_QT', max_retries => 1, retry_delay => 2, -- seconds retention_time => 60*60*24*7 -- 1w ); END; / BEGIN dbms_aqadm.start_queue( queue_name => 'ORDER_AQ', enqueue => TRUE, dequeue => TRUE ); END; /
  32. 32. gschmutz curl -X "POST" "$DOCKER_HOST_IP:8083/connectors" -H "Content-Type: application/json" --data '{ "name": "jms-source", "config": { "name": "jms-source", "connector.class": "com.datamountaineer...JMSSourceConnector", "connect.jms.initial.context.factory": "oracle.jms.AQjmsInitialContextFactory", "connect.jms.initial.context.extra.params": "db_url=jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1,java.naming.security.principal=order_p rocessing,java.naming.security.credentials=order_processing", "tasks.max": "1", "connect.jms.connection.factory": "ConnectionFactory", "connect.jms.url": "jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1", "connect.jms.kcql": "INSERT INTO order SELECT * FROM order_aq WITHTYPE QUEUE WITHCONVERTER=`com.datamountaineer.streamreactor.connect.converters.source.JsonSimpleConvert er`" } }' DB-K_5 – Setup AQ and Kafka Connect (I) Bi-directional integration between Oracle RDBMS & Apache Kafka
  33. 33. gschmutz DB-K_5: Oracle Advanced Queuing & MirrorMaker Bi-directional integration between Oracle RDBMS & Apache Kafka Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ (Kafka API) MirrorMaker Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  34. 34. gschmutzBi-directional integration between Oracle RDBMS & Apache Kafka Blueprints Apache Kafka => Oracle RDBMS
  35. 35. gschmutz Blueprints Apache Kafka => Oracle RDBMS (K-DB) Bi-directional integration between Oracle RDBMS & Apache Kafka Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry DB-K_1: Regular Polling ofRDBMS table/view DB-K_2: Regular Polling ofRDBMS API DB-K_3: Change Data Capture (CDC) on RDBMS DB-K_4: Produce Messages to Event Hub from RDBMS DB-K_5: Queuing on RDBMS to bridge to Event Hub DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5 K-DB_1: Write directly to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Queuing on RDBMS to bridge with Event Hub K_DB-1 K_DB-2 K_DB-3 K_DB-4
  36. 36. gschmutz K-DB_1: Write directly to RDBMS table/view Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  37. 37. gschmutz K-DB_1: Write to RDBMS table/view Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  38. 38. gschmutz K-DB_2: Write over RDBMS API Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  39. 39. gschmutz K-DB_2: Write over Oracle REST Database service Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  40. 40. gschmutz DB-K_2 – Setup ORDS (I) Bi-directional integration between Oracle RDBMS & Apache Kafka ORDS.DEFINE_HANDLER( p_module_name => 'customer', p_pattern => 'customer', p_method => 'POST', p_source_type => 'plsql/block', p_items_per_page => 0, p_source => 'DECLARE L_CU CLOB := :body_text; BEGIN INSERT INTO customer_t (id, first_name, last_name, title, notification_on, email, slack_handle, twitter_handle) SELECT * FROM json_table(L_CU, ''$'' COLUMNS ( id NUMBER PATH ''$.id'', first_name VARCHAR2 PATH ''$.firstName'', last_name VARCHAR2 PATH ''$.lastName'', title VARCHAR2 PATH ''$.title'', notification_on VARCHAR2 PATH ''$.notificationOn'', email VARCHAR2 PATH ''$.email'', slack_handle VARCHAR2 PATH ''$.slackHandle'', twitter_handle VARCHAR2 PATH ''$.twitterHandle'' )); INSERT INTO address_t (customer_id, id, street, nr, city, postcode, country) SELECT * FROM json_table( ... )
  41. 41. gschmutz K-DB_3: Consume from Event Hub Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  42. 42. gschmutz K-DB_3: Oracle Big Data SQL exposes topic as table Bi-directional integration between Oracle RDBMS & Apache Kafka Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  43. 43. gschmutz K-DB_4: Queuing on RDBMS to bridge with Event Hub Bi-directional integration between Oracle RDBMS & Apache Kafka Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  44. 44. gschmutz K-DB_4: Use AQ on RDBMS to bridge with Event Hub Bi-directional integration between Oracle RDBMS & Apache Kafka Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling
  45. 45. gschmutz K-DB_4: Use AQ on RDBMS to bridge with Event Hub Bi-directional integration between Oracle RDBMS & Apache Kafka Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency Tight coupling Loose Coupling Message Latency Coupling MirrorMaker AQ (Kafka API)
  46. 46. gschmutzBi-directional integration between Oracle RDBMS & Apache Kafka Demo
  47. 47. gschmutz Demo Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write directly to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Queuing on RDBMS to bridge with Event Hub K_DB-1 K_DB-2 K_DB-3 K_DB-4 DB-K_1: Regular Polling ofRDBMS table/view DB-K_2: Regular Polling ofRDBMS API DB-K_3: Change Data Capture (CDC) on RDBMS DB-K_4: Produce Messages to Event Hub from RDBMS DB-K_5: Queuing on RDBMS to bridge to Event Hub DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5 Bi-directional integration between Oracle RDBMS & Apache Kafka
  48. 48. gschmutzBi-directional integration between Oracle RDBMS & Apache Kafka Technology on its own won't help you. You need to know how to use it properly.
  • TakumiYamaoka

    Oct. 29, 2019
  • minsoojo

    May. 3, 2019
  • Khaled41

    Apr. 4, 2019
  • StreamingAnalytics

    Mar. 17, 2019

Apache Kafka is a popular distributed streaming data platform. A Kafka cluster stores streams of records (messages) in categories called topics. It is the architectural backbone of modern data analytics. Data flowing into Kafka often originates from native data streams such as social media streams, telemetry data, financial transactions and many others. But these data streams only contain part of the information. A lot of data necessary in stream processing is stored in traditional systems backed by relational databases. To implement new and modern, real-time solutions, an up-to-date view of that information is needed. So how do we make sure that information can flow between the RDMBS and Kafka, so that changes are available in Kafka as soon as possible in near-real-time? It this session, we present different approaches for integrating relational databases with Kafka, such as Kafka Connect, Oracle GoldenGate and bridging Kafka with Oracle Advanced Queuing (AQ).

Views

Total views

1,151

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

54

Shares

0

Comments

0

Likes

4

×