Oracle CloudWorld 2023 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP

A High-Speed Data Ingestion
Service in Java Using MQTT,
AMQP, and STOMP
September 20, 2023
Las Vegas, NV
Oracle CloudWorld Copyright © 2023, Oracle and/or its affiliates
1

Kuassi Mensah
Director of Product Management
Oracle
Juarez Barbosa Junior
Sr. Principal Java Developer Evangelist
Oracle
2
Meet the speakers

Java App Dev with Oracle Database
3
ORACLE JDBC & UCP

Supporting all Java DataBase Connectivity types
4
ORACLE JDBC

Reactive Java Database Connectivity (JDBC)
5
SYNC & ASYNC JDBC

Support for the Latest Java Versions
6
JAVA & JDBC VERSIONS
• Java 11 - native support, compiled with it
• Java 17 - certified
• JDBC Standards - 4.2 and 4.3
• GraalVM - native image instrumentation
• Reactive Streams - Java Flow API support
• Project Loom - Virtual Threads support
• Data access is critical in mission-critical apps

Classic Java Threads
7
AKA JAVA PLATFORM THREADS
⚫ Database access with blocking threads
⚫ A JDBC call blocks a thread for 100’s of milliseconds
(thread-per-request style)
⚫ Thread count increases linearly with the number of JDBC
calls
⚫ Performance degrades as the number of threads increases
⚫ 1 MB of stack memory per thread
⚫ Scheduling many platform threads is expensive
⚫ Preemptive scheduling vs time-slicing

Virtual Threads
8
LIGHTWEIGHT THREADS
⚫ JEPs 425, 436, 444 (JDK Enhancement Proposals)
⚫ Lightweight threads that dramatically reduce the effort of writing, maintaining, and observing high-throughput concurrent
applications
⚫ Enable applications written in the simple thread-per-request style to scale with near-optimal hardware utilization
⚫ Enable easy troubleshooting, debugging, and profiling of virtual threads with existing JDK tools

Virtual Threads
9
LIGHTWEIGHT THREADS
⚫ JEPs 425, 436, 444 (JDK Enhancement Proposals)
⚫ Do not remove the traditional implementation of threads, and do not alter the basic concurrency model of Java
⚫ Enable existing code that uses Java threads to adopt virtual threads with minimal change
⚫ java.lang.Thread
⚫ final Boolean - isVirtual()
⚫ static Thread.Builder.OfVirtual - ofVirtual()
⚫ static Thread.Builder.OfPlatform - ofPlatform()
⚫ static Thread - startVirtualThread(Runnable task)

Demo # 1: Virtual vs Platform Threads
10
RESOURCES UTILIZATION COMPARISON
⚫ Virtual Threads versus Platform Threads
⚫ JEP 425 Virtual Threads (Preview)
⚫ JEP 436 (Second Preview)
⚫ JEP 444 (Target -> JDK 21)
⚫ Runs on Java 19, 20, 21
⚫ javac --release 19 --enable-preview
⚫ java --enable-preview
⚫ Oracle JDBC Driver 23c instrumented to support Virtual Threads
⚫ Verifiable comparison of OS/HW resources utilization (Platform Threads x Virtual Threads)

Oracle Reactive Streams Ingestion (RSI) Library
11
ASYNC JDBC
⚫ Java Library for Reactive Streams Ingestion
⚫ Streaming capability: Ingest data in an unblocking, and reactive way from a
large group of clients
⚫ Group records through RAC (Real App Clusters), and Shard affinity using
native UCP (Universal Connection Pool)
⚫ Optimize CPU allocation while decoupling record processing from I/O
⚫ Fastest insert method for the Oracle Database through Direct Path Insert,
bypassing SQL and writing directly into the DB files

Demo # 2: Reactive Streams Ingestion (RSI)
PushPublisher API
12
// Push Publisher for Simple Usage
PushPublisher<Customer> pushPublisher =
ReactiveStreamsIngestion.pushPublisher();
pushPublisher.subscribe(rsi.subscriber());
// Ad-hoc usage
pushPublisher.accept(
new Customer(1, "John Doe", "North"));
// As a Consumer of a Stream
Customer[] customers = …
Stream
.of(customers)
.filter(c -> c.region.equals("NORTH"))
.forEach(pushPublisher::accept);
// As a Consumer for 3rd party Reactive Stream
// Libraries eg: Reactor https://projectreactor.io/
Flux
.just(
new Customer(1, "John Doe", "North"),
new Customer(2, "Jane Smith", "South"))
.concatWithValues(new Customer(3, "Bob", "South"))
.doOnComplete(() -> {System.out.println("Done!");})
.doOnCancel(() -> {System.out.println("Canceled!");})
.subscribe(pushPublisher::accept);

Oracle R2DBC: from Synchronous to Reactive JDBC
Sync, blocking Async, reactive
13
static String queryJdbc(java.sql.Connection connection)
throws SQLException {
try (java.sql.Statement statement =
connection.createStatement()) {
ResultSet resultSet =
statement.executeQuery("SELECT * FROM CUSTOMERS");
if (resultSet.next())
return resultSet.getString(1);
else
throw new NoSuchElementException("Query returned zero
rows");
}
}
static Publisher<String>
queryR2dbc(io.r2dbc.spi.Connection connection) {
return Flux.from(connection.createStatement(
"SELECT * FROM CUSTOMERS")
.execute())
.flatMap(result ->
result.map(row -> row.get(0, String.class)))
.switchIfEmpty(Flux.error(
new NoSuchElementException("Query returned zero
rows")));
}

Demo # 3: Oracle R2DBC
14
REACTIVE STREAMS
⚫ Oracle Reactive Relational Database Connectivity (R2DBC)
⚫ Oracle R2DBC Driver is a Java library that supports reactive Programming with Oracle
Database
⚫ It implements the R2DBC Service Provider Interface (SPI) as specified by the Reactive
Relational Database Connectivity (R2DBC)
⚫ The R2DBC SPI exposes Reactive Streams as an abstraction for Remote database
operations
⚫ The sample code uses Project Reactor. It could use RxJava, Akka, or any RS library
⚫ Runs on Java 11+

Oracle JDBC supports both!
Want Virtual Threads?
• Oracle JDBC has been “Virtual Thread Compatible”since
21.1.0.0
Want Reactive Streams?
• Oracle R2DBC 1.0.0
• Consume JDK 9 Flow interfaces implementations directly
from Oracle JDBC’s Reactive Extensions
15
Virtual Threads or Reactive Streams?

Virtual Threads or Reactive?
16
• Benefits of Virtual Threads:
• Easier to read and write
• Easier to debug
• Integration with JDK tools
• Do not alter the basic concurrency model of Java
• Limitations of Virtual Threads:
• Some libraries are not compatible
• To be finalized in JDK 21
• JEP 425: Virtual Threads (Preview)
• JEP 436: Virtual Threads (Second Preview)
• JEP 444: Virtual Threads
Benefits of Reactive:
• Reactive Libraries (Reactor, RxJava, Akka,
Vert.x)
• Stream-like API with a functional style
• Low-level concurrency is handled for
you (locks, atomics, queues).
Limitations of Reactive:
• Steep learning curve
• Harder to read and write
• Harder to debug

MOM - Message-Oriented Middleware
A middleware component that allows communication and exchanges data (messages).
It involves passing data between applications using a communication channel that carries self-
contained units of information (messages).
In a MOM-based communication environment, messages are sent and received
asynchronously.
Message-Oriented Middleware - Apache ActiveMQ
17

AMQP – Advanced Message Queuing Protocol
The Advanced Message Queuing Protocol (AMQP) is an open standard for passing business
messages between applications or organizations. With AMQP organizations can:
• Realize the savings commoditization brings; remove vendor lock-in
• Connect applications on different platforms; choose the right platform for the job
• Connect to business partners using a full-featured open standard; remove technical
barriers to trade
• Position for innovations built upon the foundation of AMQP
• Security, Reliability, Interoperability, Standard, Openness
18

MQTT – Message Queuing Telemetry Transport
MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT).
It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal
for connecting remote devices with a small code footprint and minimal network bandwidth.
19

STOMP – Simple (or Streaming) Text Oriented
Messaging Protocol
STOMP provides an interoperable wire format so that STOMP clients can communicate with
any STOMP message broker to provide easy and widespread messaging interoperability
among many languages, platforms and brokers..
20

Java Message Service (JMS) & ActiveMQ
ActiveMQ
• ActiveMQ is a Java-based open-source message broker which supports REST API and various
wire-level protocols, such as MQTT, AMQP, and STOMP.
JMS Message Producer and Consumer
• Port number 5672 is the default port that ActiveMQ is listening to over the AMQP protocol when it
starts up.
• The sample code snippet creates an AMQP connection (publisher) that connects to ActiveMQ
using the given username, password, hostname, and port number.
• It also creates a topic subscriber (consumer) to receive messages that have been published to the
"event" topic.
21

Java Message Service (JMS) & ActiveMQ
22

Demo # 4: Architecture / Project Structure
ActiveMQ + AMQP + MQTT + STOMP + Java Message Service
23

Learn More
JDK 21 - https://openjdk.org/projects/jdk/21/
Project Loom - https://openjdk.org/projects/loom/
JEP 444 Virtual Threads - https://openjdk.org/jeps/444
Getting Started with the Java library for Reactive Streams Ingestion (RSI) - https://bit.ly/3rEiRnC
High-throughput stream processing with the Java Library for Reactive Streams Ingestion (RSI),
Virtual Threads, and the Oracle ATP Database - https://bit.ly/3rATCTd
Oracle R2DBC Driver – https://github.com/oracle/oracle-r2dbc
JDBC – https://www.oracle.com/database/technologies/appdev/jdbc.html
Application Development with the Oracle Database -
https://www.oracle.com/database/technologies/application-development.html
24

Thank you
Kuassi Mensah, Director of Product Management
kuassi.mensah@oracle.com
25
Juarez Barbosa Jr, Sr. Principal Java Developer Evangelist
juarez.barbosa@oracle.com

Oracle CloudWorld 2023 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP

Recommended

Recommended

More Related Content

Similar to Oracle CloudWorld 2023 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP

Similar to Oracle CloudWorld 2023 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP (20)

More from Juarez Junior

More from Juarez Junior (14)

Recently uploaded

Recently uploaded (20)

Oracle CloudWorld 2023 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP