Alluxio Product School Webinar May 24. 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Shouwei Chen (core maintainer & product manager, Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for robust data infrastructures tailored for large language model (LLM) training and inference in the cloud. To effectively leverage the breakthroughs in LLM, organizations must ensure low latency, high concurrency, and scalability in production environments. In this Alluxio-hosted webinar, Shouwei will present on the design and implementation of a distributed caching system that addresses the I/O challenges of LLM training and inference. He will explore the unique requirements of data access patterns and offer practical best practices for optimizing the data pipeline through distributed caching in the cloud. The session will feature insights from real-world examples, such as Microsoft, Tencent, and Zhihu, as well as from the open-source community. Attendees will leave with a deeper understanding of how to harness scalable, efficient, and robust data infrastructures for LLM training and inference.

1. Distributed Caching for
Generative AI: Optimizing
LLM Data Pipelines
Shouwei Chen @Alluxio

2. Head of Community,
Developer Relations
@jasminechenwang
Core Maintainer & PM
Shouwei Chen

3. Join the conversation
on Slack
alluxio.io/slack
1,200+ Github
contributors & growing
10,000+ Slack
Community Members
Top 10 Most Critical Java
Based Open Source Project[1]
GitHubʼs Top 100 Most Valuable
Repositories Out of 96 Million
[1] Google Comes Up With A Metric For Gauging Critical Open-Source Projects
Open Source Started From UC Berkeley AMPLab

4. Analytics & AI
in the Hybrid & Multi-Cloud Era
Available:

6. I/O challenges In ML/DL
Metadata Scalability
CV Training data often
consists of a massive
amount of small raw
files (billions of 100KB
photos)
Cloud API cost
Training job can read
ten thousands of
times for single copy of
files, result in
thousands $
cost/training
High IO throughput
Training jobs are highly
concurrent, require
high I/O to keep
GPU&CPU utilized

7. Using Alluxio For DL
Distributed Caching
70
70 70
70 70
70
POSIX POSIX POSIX
▪ No full data copy from data source
▪ Serve fast DL training without HPC hardware
▪ Full data management for your machine learning unstructured datasets

8. Online data platform
Offline data platform / Data warehouse
Inference cluster
models
Offline training platform
Training cluster
Training data
Models /
Training data
models
Training data
Training cluster
models
training data
training data
Training data
Architecture diagram w/ Alluxio as access layer

9. Online data platform
Offline data platform / Data warehouse
Inference cluster
models
Offline training platform
Training cluster
Training data
Models /
Training data
models
Training data
Training cluster
models
training data
training data
Training data
Problem solved - compare to data migration
Expensive HPC hardware
Customization on data migration create overhead for eng efforts
Engineer have to manually delete the outdate data in persistent layer(Cloud storage & HDFS)
User have to understand the data pipeline between data sources & AI/ML infra
Problem solved - compare to direct access
Low GPU Utilization
Cloud storage API cost in same region / across region data movement cost
Coupled architecture - Business driven(Vendor lock-in)
Architecture diagram w/ Alluxio as access layer

10. High Scalability
Tens of Billions files /
Training
ESSENTIAL
High Availability
99.99%
ESSENTIAL
High Performance
Higher GPU utilization
ESSENTIAL
Always increasing expectations…

11. Availability
● No single point failure
● Fault tolerance
● More friendly to K8S and Cloud (No statefulset
needed)
Performance & Scalability
● Unlimited scalability
● Support billions of small files
● Streamlined RPC call
Multi-Tenant in One Cluster
● Proven effectiveness of cost
● Tenant isolation
● Serverless interface
● Easy management
Plugable Quota and Security Management
● Customizable quota management
● Token passthrough for S3/GCS/OSS
● Impersonation & Krb5 for HDFS
Motivation & Benefits

12. Dora Core Components - Affinity
● Decide which worker to go on
client side
● Worker handle
list/getFileStatus call
● Fetch the worker list from
service periodically.
● Worker persists the metadata
to support quota and tenant
isolation
● For the unsupported calls or
when worker is down, clients
can fallback to UFS

13. Performance Improvement by Netty
Use Netty for Data Transmission by configuration.
Dora provide netty data transmission that improves reading
performance by 30%-50%.
Advantages compared to gRPC:
● Less data copies through different thread pools.
● Zero-copy transmission avoiding serialization of
Protobuf.
● Avoid OOM by optimizing off-heap memory usage.
● Requires less bytes to transfer as there is no additional
HTTP header any more.

14. IO throughput improvements
● Up to 9X for unstructured data
● 2-15X for structured data
High Concurrent Read in Dora

15. Twitter.com/alluxio
Linkedin.com/alluxio
Website
www.alluxio.io
Slack
https://alluxio.io/slack
@
Social Media
Github
https://github.com/Alluxio