Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo Apache NiFi - Cloudera DataFlow Apache Flink - Cloudera Streaming Analytics Apache Kafka - Streaming LLM / WatsonX.AI Generative AI granite and llama2 models slack to slack apps

1. November 2, 2023 | 9:00 AM – 6:00 PM
INTEGRATING AI INTO REAL-TIME DATA
PIPELINES

2. Integrating AI Into Real-Time Data
Pipelines

3. 3
© 2023 Cloudera, Inc. All rights reserved.
Streaming
data
Data at rest
Change data
capture
Any
DATA
Real-time
Processing
• Analyze data in
motion
• Continuous
monitoring
• Trends and
anomalies
Data
Lakehouse
Data
products
Any
BUSINESS
EVENT
Continuous
Results
• No-Code UI
• Author once
publish anywhere
• Analytics lifecycle
management for
dev/ops
Any
DATA
ANALYST
AI models
Event-driven
apps
Analytics apps
Any
DATA
CONSUMER
Data Relevance for Real-Time Applications

4. 4
© 2023 Cloudera, Inc. All rights reserved.
INGEST PREPARE PUBLISH
DATA SOURCES
Internal Users
(After Sales)
External
Systems
ENTERPRISE
LAKEHOUSE
CAPABILITY VIEW
INGESTION
MESSAGE HUB
STORAGE
BATCH
MANAGEMENT
STREAM
CONSUMPTION
Closed Loop
Systems
SQL Stream Builder
Machine Learning
Data Visualization
Workload Manager
watsonx.data

5. 5
© 2023 Cloudera, Inc. All rights reserved.
Cloudera’s Data in Motion Services
Cloudera Offers Two Core Data-In-Motion Services: DataFlow & Stream Processing
DATAFLOW — Powered by Apache NiFi, it enables
developers to connect to any data source anywhere with
any structure, process it, and deliver to any destination
using a low-code authoring experience.
CLOUDERA SDX — Secure, Monitor and Govern your
Streaming workloads with the same tooling using Apache
Ranger & Apache Atlas.
STREAM PROCESSING — Powered by Apache Flink and
Kafka, it provides a complete, enterprise-grade stream
management and stateful processing solution. With
support for industry standard interfaces like SQL,
developers, data analysts, and data scientist can easily
build a wide variety of hybrid real-time applications.

6. 6
© 2023 Cloudera, Inc. All rights reserved.
Simplified Streaming Pipelines
Connect to any data source anywhere, process and deliver to any destination
Ingest Process Distribute
Active
Passive
Route
Filter
Enrich
Transform
Data born in
the cloud
Data born
outside the
cloud
Any
destination
Connectors
Gateway
Endpoint
Connect & Pull
Send
Connectors
Deliver

7. LLM USE CASE
Vector DB
AI Model
Unstructured file types
Data in Motion
on Cloudera Data
Platform (CDP)
Capture, process &
distribute any data,
anywhere
Other enterprise data Open Data Lakehouse
Materialized Views
Structured Sources
Applications/API’s
Streams

8. 8
© 2023 Cloudera, Inc. All rights reserved.
Apache NiFi in a few numbers
A very active project with a dynamic community & comparison with ACEU 2019
2800+ members on the Slack channel (535+ - 4 years ago)
475+ contributors on Github across the repositories (260+ - 4 years ago)
65 committers in the Apache NiFi community (45 - 4 years ago)
Apache NiFi 1.23.2 is the latest release, NiFi 2.0 coming soon (NiFi 1.10 - 4 years ago)
14M+ docker pulls of the Apache NiFi image (1M+ - 4 years ago)