Alluxio Day III April 27, 2021 Speakers: Lu Qiu & Bin Fan, Alluxio

1. Alluxio Data Orchestration for
Machine Learning
Lu Qiu, Bin Fan @ Alluxio
04/27/2021
1

2. About Us – Lu Qiu
● Software Engineer @ Alluxio
● Email: lu@alluxio.com
● Master Data Science @ GWU
● Areas: Alluxio fault tolerant system, journal
system, metrics system, and POSIX API.
Alluxio integration with Cloud
2

3. About Us – Bin Fan
● Founding Engineer, VP Open Source @ Alluxio
● Email: binfan@alluxio.com
● PhD in CS @ CMU
3

4. Agenda
● What is Alluxio POSIX API
● How to Use Alluxio via POSIX API
● Latest Work and Roadmap
4

5. What is Alluxio POSIX API
5

6. What is POSIX？
https://en.wikipedia.org/wiki/POSIX
- Portable Operating System Interface
- Define API, command line shells, utility interfaces for software
compatibility with variants of Unix and other operating systems
- Maintaining compatibility between operating systems
- A standard makes things stay compatible in operating systems
6

7. Apps Connecting to Alluxio via POSIX API
7

8. Accessing Remote/Distributed Data as
Local Directories
8
HDFS #1
Obj Store
NFS
HDFS #2
Connecting to
• HDFS
• Amazon S3
• Azure
• Google Cloud
• Ceph
• NFS
• Many more

9. Alluxio
Server
Alluxio
Server
Model Training
Distributed Caching w/ Unified Namespace
Alluxio
Server
A
B
/path1/file1
/path2/file2
C
A
B C A
Model Training Model Training
9

10. Under the Hood: FUSE
https://en.wikipedia.org/wiki/Filesystem_in_Userspace
- Filesystem in Userspace
- A software interface for Unix and Unix-like computer operating systems
that lets non-privileged users create their own file systems without editing
kernel code.
10

11. Under the Hood: FUSE (Cont.)
The userspace side of FUSE, the libfuse library
https://github.com/libfuse/libfuse
A FUSE file system is typically implemented as a standalone application that
links with libfuse.
https://github.com/libfuse/libfuse/blob/master/example/hello.c
- Define read/write/ls/…
11

12. 12
Alluxio-FUSE limitations
3/25/19
● Since Alluxio as a write-once/read-many file system, the mounted
file system will not support all POSIX workloads.
Files can be written only once, only sequentially, and never be
modified. Vim command is not supported since it uses append
internally. Cp when destination file exists will fail.
● Alluxio does not have hard-link and soft-link concepts, so the
commands like ln are not supported, neither the hardlinks number
is displayed in ll output.
● Performance is worse than using Alluxio Java client directly

13. Limitations of Alluxio POSIX API
13
● Since Alluxio as a write-once/read-many file system, the mounted file
system will not support all POSIX workloads.
Files can be written only once, only sequentially, and never be modified.
Vim command is not supported since it uses append internally. Cp when
destination file exists will fail.
● Alluxio does not have hard-link and soft-link concepts, so the commands
like ln are not supported, neither the hardlinks number is displayed in ll
output.
● Performance is bound by FUSE and Alluxio client

14. How to Use Alluxio POSIX API
14

15. Launching Standalone Fuse
15
Mount Alluxio service as a local FS path:
Check out local Alluxio mount points
Unmount Alluxio service:
integration/fuse/bin/alluxio-fuse mount
-o [mount_options] mount_point [alluxio_path]
integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
80846 /mnt/people /people
80847 /mnt/sales /sales
integration/fuse/bin/alluxio-fuse unmount mount_point

16. Bash
Tensorflow
cat /mnt/alluxio/myInput
Accessing Alluxio Service via POSIX API
16
python classify_image.py --model_dir /mnt/fuse/imagenet/

17. Demo
17

18. A New JNI-based FUSE Impl
(available since 2.5.0)
18

19. Integrating libfuse (in C) with Java Client
19
● Previously based on 3rd party JNR-based FUSE library
● Now on a new 1st party JNI-based FUSE library
○ On libfuse directly to enable more optimizations
○ Close to native libfuse performance
○ Support high concurrency

20. JNR-FUSE Hard to debug
20
● JNR-FUSE has many dependencies, hard to debug and fix.
● Didn’t support callback functions well. When a native thread call JVM, it will
attach to JVM which is relatively expensive.

21. Community Collaboration
● Community-driven collaboration
○ Contributors from NJU, Alibaba, Tencent, Alluxio
● Already in used by Microsoft in Production
21

22. Performance
22

23. Target Scenarios
23
● Multi-node, multi-thread machine learning/deep learning workloads.
● Read path has better performance benefits compared to write path
● Medium to large files have better performance than small files

24. Local RPC Elimination
(available soon in 2.6.0)
24

25. Idea:
● Motivated by training workloads reading many small files
○ Standalone Alluxio-FUSE process is a long-running client translating
FUSE API calls to Alluxio client RPCs
○ RPCs required to communicate with workers, even on cache hit
● Combining Alluxio-FUSE functionality into Alluxio worker
25

26. Launching Fuse on Worker
26
● Configure alluxio-site.properties on worker nodes:
alluxio.worker.fuse.enabled=true
alluxio.worker.fuse.mount.point=/mnt/alluxio-service
alluxio.worker.fuse.mount.options=kernel_cache,entry_timeout=7200,attr_ti
meout=7200
● Then Start Worker Process through, Alluxio namespace can be accessed
via a local path /mnt/alluxio-service

27. Other Optimizations
● DONE: Moduliazed JNI-Fuse library (github repo)
● TODO: Optimize gRPC performance on remote cache hit
● TODO: Support libfuse 3.x (issue ticket)
● And many more coming..
Join Alluxio weekly community sync to create solutions together!
27

28. Reference
● Using Alluxio to Optimize and Improve Performance of
Kubernetes-Based Deep Learning in the Cloud (link)
● ALLUXIO POSIX API documentation (English or Chinese)
● Turn Cloud Storage or HDFS IntoYour Local File System for
Faster AI Model Training With TensorFlow (link)
● Fuse realization theory (Chinese TBT link)
28

29. Questions?
Welcome to join the Alluxio Community!
www.alluxio.io/slack | @alluxio
29