Real-Time AI Streaming - AI Max Princeton

© 2024 Cloudera, Inc. All rights reserved.
Building Real-Time Generative AI
Pipelines
Tim Spann
Principal Developer Advocate
April 12, 2024
AI Max
Summit

© 2024 Cloudera, Inc. All rights reserved. 3
This week in Apache NiFi, Apache Flink,
Apache Kafka, ML, AI, Apache Spark, Apache
Iceberg, Python, Java and Open Source
friends.
https://bit.ly/32dAJft
https://www.meetup.com/futureofdata-
princeton/
FLaNK Stack Weekly by Tim Spann

Conﬁdential—Restricted
@PaasDev
https://www.meetup.com/futureofdata-princeton/
From Big Data to AI to Streaming to Containers to
Cloud to Analytics to Cloud Storage to Fast Data to
Machine Learning to Microservices to ...
Future of Data - NYC + NJ + Philly + Virtual
https://linktr.ee/tspannhw

Tim Spann
Twitter: @PaasDev Blog: datainmotion.dev
Principal Developer Advocate
Princeton Future of Data Meetup
ex-Pivotal, ex-Hortonworks,
ex-StreamNative, ex-HPE,
ex-PwC, ex-EY.
https://medium.com/@tspann
https://github.com/tspannhw

Some common Vector DBs Open Community & Open Models
RAPID INNOVATION IN THE LLM SPACE
Too much to cover today.. but you should know the common LLMs, Frameworks, Tools
Notable LLMs
Closed Models Open Models
GPT3.5
GPT4
Llama2
Mistral7B
Mixtral8x7B
Claude2
++ 100s more… check out the HuggingFace LLM
Leaderboard (pretrained, domain fine-tuned, chat models, …)
Code Llama
Popular LLM Frameworks
When to use one over the other? Use Langchain if you need a general-purpose framework with
flexibility and extensibility. Consider LlamaIndex if you’re building a RAG only app (retrieval/search)
Langchain is a framework for
developing apps powered by LLMs
● Python and JavaScript Libraries
● Provides modules for LLM
Interface, Retrieval, & Agents
LLamaIndex is a framework designed
specifically for RAG apps
● Python and JavaScript Libraries
● Provides built in optimizations /
techniques for advanced RAG
HuggingFace is an ML community for hosting &
collaborating on models, datasets, and ML applications
● Latest open source LLMs are in HuggingFace
● + great learning resources / demos
https://huggingface.co/
Open Source vs Self Hosted vs SaaS option

Enterprise Knowledge Base / Chatbot / Q&A
- Customer Support & Troubleshooting
- Enable open ended conversations with user
provided prompts
Code assistant:
- Provide relevant snippets of code as a
response to a request written in natural
language.
- Assist with creating test cases and
synthetic test data.
- Reference other relevant data such as a
company’s documentation to help provide
more accurate responses.
Social and emotional sensing
- Gauge emotions and opinions based on a
piece of text.
- Understand and deliver a more nuanced
message back based on sentiment.
ENTERPRISE WIDE USE CASES FOR AN LLM
Classiﬁcation and Clustering
- Categorize and sort large volumes of data
into common themes and trends to support
more informed decision making.
Language Translation
- Globalize your content by feeding web
pages through LLMs for translation.
- Combine with chatbots to provide
multilingual support to your customer base.
Document Summarization
- Distill large amounts of text down to the
most relevant points.
Content Generation
- Provide detailed and contextually relevant
prompts to develop outlines, brainstorm
ideas and approaches for content.
L
Adoption dependent upon an Enterprise’s risk tolerance, restrictions, decision rights and disclosure obligations.

NLP / AI / LLM
Generative AI

9
Which Model and When?
Use the right model for right job: closed or open-source
Closed
Source
Usage can
easily scale
but so can
your costs
Rapidly
improving
AI models
Most
advanced
AI models
Excel at more
specialized
tasks
Great for a
wide range
of tasks
Open
Source
Better cost
planning
Compliance,
privacy, and
security risks
More control
over where &
how models
are deployed

10
Adoption of Generative AI is a Journey
Identifying AI challenges in the enterprise
Data integration
barriers
● Streamlined access to
enterprise data
Rigid model
infrastructure
● Modularity
● Flexibility
● AI Ops
Lack of security
and transparency
● Model control
● Built-in security
● Visibility & governance
What’s missing
Challenges

11
Data = Organization Context
Your data enables contextually accurate responses from LLMs
Large Language
Model
User Query
Contextually
Inaccurate
Response
Data
Organization
Context
User Query
Large Language
Model
Contextually
Accurate
Response

CLOSED-SOURCE
FOUNDATION MODELS
MODEL HUBS
OPEN SOURCE
FOUNDATION MODELS
FINE-TUNED MODELS
PRIVATE
VECTOR STORE
MANAGED
VECTOR STORE
CLOUD INFRASTRUCTURE
Milvus, Solr*
Meta (Llama 2)
Applied Machine Learning Prototypes (AMPs)
Hugging Face
Pinecone
SPECIALIZED HARDWARE
APIs: OpenAI (GPT-4 Turbo)
Amazon Bedrock: Anthropic (Claude 2), Cohere…
DATA
WRANGLING
REAL-TIME
DATA INGEST
& ROUTING
AI MODEL
TRAINING &
INFERENCE
DATA STORE &
VISUALIZATION
Open Data Lakehouse
DATA
WRANGLING
REAL-TIME
DATA INGEST
& ROUTING
AI MODEL
TRAINING &
SERVING
DATA STORE &
VISUALIZATION
AI APPLICATIONS

Live Q&A
Travel Advisories
Weather Reports
Documents
Social Media
Databases
Transactions
Public Data Feeds
S3 / Files
Logs
ATM Data
Live Chat
…
ARCHITECTURE
INTERACT
COLLECT STORE
ENRICH, REPORT
Distribute
Collect
Report
REPORT
Visualize
Report, Automate
AI BASED ENHANCEMENTS
Predict, Automate
VECTOR DATABASE
LLM
Machine
Learning
Data
Visualization
Data Flow
Data
Warehouse
SQL
Stream Builder
Data
Visualization
Input Sentences
Generated Text
Timestamp
Input Sentence
Timestamps
Enrichments
Messaging
Broker
Real-time alerting
Real-time alerting
Aggregations

DATAFLOW / STREAMING

LLM USE CASE
Vector DB
AI Model
Unstructured ﬁle types
Data in Motion
on Cloudera Data
Platform (CDP)
Capture, process &
distribute any data,
anywhere
Other enterprise data Open Data Lakehouse
Materialized Views
Structured Sources
Applications/API’s
Streams

https://medium.com/cloudera-inc/getting-ready-for-apache-nifi-2-0-5a5e6a67f450
NiFi 2.0.0 Features
● Python Integration
● Parameters
● JDK 21+
● JSON Flow Serialization
● Rules Engine for Development
Assistance
● Run Process Group as Stateless
● ﬂow.json.gz
https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Release+Goals

17
DataFlow Pipelines Can Help
External Context Ingest
Ingesting, routing, clean, enrich, transforming,
parsing, chunking and vectorizing structured,
unstructured, semistructured, binary data and
documents
Prompt engineering
Crafting and structuring queries to optimize
LLM responses
Context Retrieval
Enhancing LLM with external context such as
Retrieval Augmented Generation (RAG)
Roundtrip Interface
Act as a Discord, REST, Kafka, SQL, Slack bot to
roundtrip discussions

18
UNSTRUCTURED DATA WITH NIFI
• Archives - tar, gzipped, zipped, …
• Images - PNG, JPG, GIF, BMP, …
• Documents - HTML, Markdown, RSS, PDF, Doc, RTF, Plain Text, …
• Videos - MP4, Clips, Mov, Youtube URL…
• Sound - MP3, …
• Social / Chat - Slack, Discord, Twitter, REST, Email, …
• Identify Mime Types, Chunk Documents, Store to Vector Database
• Parse Documents - HTML, Markdown, PDF, Word, Excel, Powerpoint

19
CLOUD ML/DL/AI/Vector Database Services
• Cloudera ML
• Amazon Polly, Translate, Textract, Transcribe, Bedrock, …
• Hugging Face
• IBM Watson X.AI
• Vector Stores Anywhere: Weaviate, Pinecone, Milvus,
Chroma DB, SOLR, …

20
https://medium.com/cloudera-inc/google-gemma-for-real-time-lightweight-open-llm-infe
rence-88efe98e580f

Python Processors

Extract Entities
● Python 3.10+
● NLP, SpaCY
● Extract locations
● Extract organizations
● Extract money
● Extract time
● Extract events
● Extract countries
● Extract objects, food, people, quantities
https://github.com/tspannhw/FLaNK-python-processors/blob/main/ExtractEntities.py

Extract Company Names
● Python 3.10+
● Hugging Face, NLP, SpaCY, PyTorch
https://github.com/tspannhw/FLaNK-python-ExtractCompanyName-processor

WatsonX SDK To Foundation
● Python 3.10+
● LLM
● WatsonX.AI Foundation Models
● Inference
● Secure
● Oﬃcial SDK from IBM
https://github.com/tspannhw/FLaNK-python-watsonx-processor

CaptionImage
● Python 3.10+
● Hugging Face
● Salesforce/blip-image-captioning-large
● Generate Captions for Images
● Adds captions to FlowFile Attributes
● Does not require download or copies of
your images
https://github.com/tspannhw/FLaNK-python-processors

RESNetImageClassification
● Python 3.10+
● Hugging Face
● Transformers
● Pytorch
● Datasets
● microsoft/resnet-50
● Adds classiﬁcation label to FlowFile
Attributes
your images

NSFWImageDetection
● Python 3.10+
● Hugging Face
● Transformers
● Falconsai/nsfw_image_detection
● Adds normal and nsfw to FlowFile
Attributes
● Gives score on safety of image
your images

FacialEmotionsImageDetection
● Python 3.10+
● Hugging Face
● Transformers
● facial_emotions_image_detection
● Image Classiﬁcation
● Adds labels/scores to FlowFile Attributes
your images

Other Python Processors
● Chunk Document, Parse Document
● Prompt Chat GPT
● Put Chroma, Query Chroma
● Put Pinecone, Query Pinecone

FLINK SQL

FLINK SQL -> CLOUDERA MACHINE LEARNING MODELS

FLINK SQL -> NIFI -> HUGGING FACE GOOGLE GEMINI

SSB UDF JS/JAVA + GenAI = Real-Time GenAI SQL
https://medium.com/cloudera-inc/adding-generative-ai-results-to-sql-streams-513e1fd2a6af
SELECT CALLLLM(CAST(messagetext as
STRING)) as generatedtext,
messagerealname, messageusername,
messagetext,messageusertz,
messageid, threadts, ts
FROM flankslackmessages
WHERE messagetype = 'message'

SSB MATERIALIZED VIEWS
Key Takeaway; MV’s allow data scientist, analyst and developers consume data from the ﬁrehose

© Cloudera, Inc. All rights reserved.
Apache Flink SQL
Democratize access to real-time data with just SQL

Infer Tables from Kafka Topics with JSON or Avro

APACHE KAFKA

Let’s do a metamorphosis on your data. Don’t fear changing data.
You don’t need to be a brilliant writer to stream
data.
Franz Kafka was a German-speaking
Bohemian novelist and short-story writer,
widely regarded as one of the major figures of
20th-century literature. His work fuses
elements of realism and the fantastic.
Wikipedia
YES, FRANZ, IT’S KAFKA

Streams
Replication
Manager (SRM)
• Event Replication engine for Kafka
• Supports active-active, multi-cluster,
cross DC replication scenarios
• Leverage Kafka Connect for
scalability and HA
• Replicate data and conﬁgurations
(ACL, partitioning, new topics, etc)
• Offset translation for simpliﬁed
failover
• Integrate replication monitoring with
SMM

APACHE ICEBERG

Cloudera’s Open Data Lakehouse
❏ Multi-function analytics for Streaming, Data
Engineering, Data Warehouse and AI/ML with
integrated data services
❏ Common security and governance policies and data
lineage with SDX integration
❏ Common dataset with all CDP analytics engines
without data duplication and movement
❏ Deployment freedom with Multi-Hybrid Cloud
Iceberg Tables
DATA
WAREHOUSE
MACHINE
LEARNING
DATA
ENGINEERING
DATA
FLOW
STREAM
PROCESSING
Multi-Hybrid Cloud
Metadata | Security | Encryption | Control | Governance

Compute Engine Interoperability & SDX
Integration
● Snapshot isolation ensures consistent data
access and processing with various
compute engines including Hive, Spark,
Impala and Niﬁ
● Security & Governance support (e.g. FGAC)
through Ranger integration
● Data lineage support through Atlas
integration
Apache Impala
Iceberg Tables
Ranger Atlas

FLINK & ICEBERG INTEGRATION
Robust Next Generation Architecture for Data Driven Business
Uniﬁed Processing Engine Massive Open table format
Iceberg Support for Flink APIs through SSB
• Maximally open
• Maximally ﬂexible
• Ultra high performance for MASSIVE data
• Can be used as Source and Sink
• Supports batch and streaming modes
• Supports time travel

NIFI & ICEBERG INTEGRATION
• PutIceberg processor in CFM 2.1.6
• PutIcebergCDC

DEMO
I Can Haz
Data?

CSP Community Edition
● Docker compose ﬁle of CSP to run from command line w/o any
dependencies, including Flink, SQL Stream Builder, Kafka, Kafka
Connect, Streams Messaging Manager and Schema Registry.
○ $>docker compose up
● Licensed under the Cloudera Community License
● Unsupported Commercially (Community Help - Ask Tim)
● Community Group Hub for CSP
● Find it on docs.cloudera.com (see QR Code)
● Kafka, Kafka Connect, SMM, SR, Flink, Flink SQL, MV, Postgresql, SSB
● Develop apps locally

Open Source Edition
• Apache NiFi in Docker
• Try new features
quickly
• Develop applications
locally
● Docker NiFi
○ docker run --name nifi -p 8443:8443 -d -e
SINGLE_USER_CREDENTIALS_USERNAME=admin -e
SINGLE_USER_CREDENTIALS_PASSWORD=ctsBtRBKHRAx69EqUghv
vgEvjnaLjFEB apache/nifi:latest
● Licensed under the ASF License
● Unsupported
● NiFi 1.25 and NiFi 2.0.0-M2
https://hub.docker.com/r/apache/niﬁ

https://medium.com/cloudera-inc/streaming-street-cams-to-yolo-v8-with-python-and-nifi-to-minio-s3-3277e73723ce
Street Cameras

CEM, CDF, CSP
● https://medium.com/cloudera-inc/watching-airport-traffic-in-real-time-32c522a6e386
● https://medium.com/cloudera-inc/building-a-real-time-data-pipeline-a-comprehensive-tutorial-on-mi
nifi-nifi-kafka-and-flink-ee03ee6722cb
● https://medium.com/cloudera-inc/finding-the-best-way-around-7491c76ca4cb
● https://medium.com/cloudera-inc/nyc-traffic-are-you-kidding-me-6d3fa853903b
CDF
● https://medium.com/@tspann/building-a-travel-advisory-app-with-apache-nifi-in-k8-969b44c84958
LLM, GenAI, HuggingFace, WatsonX, OLLAMA, Mistral, NiFi, Python, Slack, Pytorch
● https://medium.com/@tspann/using-ollama-with-mistral-and-apache-nifi-720c17f5ff12
● https://medium.com/cloudera-inc/google-gemma-for-real-time-lightweight-open-llm-inference-88efe
98e580f
● https://medium.com/@tspann/image-processing-with-custom-python-and-nifi-2-0-06eadc62c03c
● https://medium.com/@tspann/ai-augmented-devrel-part-1-4058af905a89
● https://medium.com/cloudera-inc/mixtral-generative-sparse-mixture-of-experts-in-dataflows-59744f
7d28a9
● https://medium.com/@tspann/building-an-llm-bot-for-meetups-and-conference-interactivity-c211ea6
e3b61
● https://medium.com/@tspann/yet-another-python-processor-45aaae6fe406

https://github.com/tspannhw/PaK-Stocks
https://github.com/tspannhw/FLaNK-Py-Stocks
https://medium.com/cloudera-inc/let-nifi-worry-about-those-stoc
ks-for-you-57d5f16b5e6b

CLOUDERA STREAM PROCESSING
Two Major Capabilities: Enterprise Messaging and Powerful Stream Processing
Enterprise grade messaging products for Apache
Kafka. Streams Messaging Manager to
monitor/operate clusters, Streams Replication
Manager for HA/DR deployments, Schema Registry for
centralized schema management, and support for
Kafka Connect and Cruise Control
Cloudera Streaming Analytics (CSA)
Powered By Apache Flink
Cloudera Streams Messaging (CSM)
Powered by Apache Kafka
Powered by Apache Flink with SQL StreamBuilder, it
provides low-latency stream processing capabilities
with advanced windowing & state management made
simple with SQL

ENTERPRISE MANAGEMENT CAPABILITIES FOR APACHE KAFKA
Extend streams messaging services for Schema Mgmt, Replication & Monitoring
Schema Registry
Kafka Schema Governance
Streams Replication Manager
Kafka Replication Service
for Disaster Recovery
Streams Messaging Manager
Management & Monitoring Service
for all of your Kafka clusters

Kafka Data Movement, Operations and Security Made Easier
ENTERPRISE MANAGEMENT CAPABILITIES FOR APACHE KAFKA
Kafka Connect Support
Simple Data Movement
Change Data Capture Connectors
Build Custom Connectors with NiFi
Ranger Security
Improved ACL and Audit for
Kafka, KConnect and Schema
Registry
Cruise Control Support
Intelligent Rebalancing
& Self-Healing of your
Kafka Clusters

NEXT GENERATION STREAMING ANALYTICS WITH APACHE
FLINK
Low latency stateful stream processing
● Flink is a distributed data processing
systems ideally suited for real-time, event
driven applications.
● Uniﬁes stream and batch processing
● Advanced features - late arriving data,
checkpointing, event time processing,
Exactly Once Processing
Real-Time
Insights
Event
Processing
Low
Latency

SQL STREAM BUILDER (SSB)
SQL STREAM BUILDER allows
developers, analysts, and data
scientists to write streaming
applications with industry
standard SQL.
No Java or Scala code
development required.
Simpliﬁes access to data in Kafka
& Flink. Connectors to batch data in
HDFS, Kudu, Hive, S3, JDBC, CDC
and more
Enrich streaming data with batch
data in a single tool
Democratize access to real-time data with just SQL

LLMs ARE FOUNDATION MODELS
Base models that can be adapted for a wide range of use cases
Terabytes of Data
(Multiple Formats)
Foundation Models
(Billions of Parameters)
Train Adapt
Question/Answering
Sentiment Analysis
Doc summarization
… ++ more
➔ Historically, data scientists trained specialized models against narrow datasets to solve speciﬁc tasks.
➔ LLMs are Foundation models that can be adapted to perform a variety of tasks.
◆ It is faster to “adapt” a foundation model than it is to train a specialized model from scratch
◆ Decouples “knowledge” from “intelligence”
◆ Opens up AI use cases to software developers (instead of just specialised data scientists)

TH N Y U

Real-Time AI Streaming - AI Max Princeton

Recommended

Recommended

More Related Content

Similar to Real-Time AI Streaming - AI Max Princeton

Similar to Real-Time AI Streaming - AI Max Princeton (20)

More from Timothy Spann

More from Timothy Spann (20)

Recently uploaded

Recently uploaded (20)

Real-Time AI Streaming - AI Max Princeton