Comparison between traditional filesystems and HDFS writes

1. February 2012
Filesystems, RPC and HDFS
Alexander Lorenz

2. Agenda
1 Linux Kernel I/O Scheduler
2 I/O Stack in Linux
3 VFS Implementation
4 NFS RFC Model
5 RPC
6 HDFS
7 Limitations / Problems (Discussion)
2 ©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.

3. Linux Kernel I/O Scheduler
• Disk seek is the slowest operation in a computer
• I/O scheduler arranges the disk head to move in a
single direction to minimize seeks
• Prevent Starvation
• Improve overall disk throughput by
• Reorder requests to reduce the disk seek time
• Merge requests to reduce the number of requests
3

4. Kernel I/O Scheduler Framework
Block layer
• Linux elevator is an abstract layer to which different I/O
scheduler can attach
• Merging mechanisms are provided by request queues
enqueue • Front or back merge of a request and a bio
Merge, sort IO Scheduler • Merge two requests
Internal queues
• Sorting policy and merge decision are done in elevators
• Pick up a request to be merged with a bio
prioritize • Add a new request to the request queue
dequeue • Select next request to be processed by block drivers
external queue device driver
4

5. I/O Stack in Linux
Application
Userland
Sys Calls
Kernelspace
Filesystem
Access Locking
Cache
Prefetch Flush
Bulk writes
Meta
Disk Layout
Data
HDD Driver
5

6. VFS Implementation
Application
Userland
Sys Calls
Kernelspace
VFS
ext3 ext2 NFS CIFS
6

7. NFS RFC Model
Application
with
NFS Access
Filesystem Filesystem
NFS Client NFS Server
File Handler
RPC RPC
Kernelspace TCP/IP TCP/IP Kernelspace
UDP UDP
Local HDD Local HDD
7

8. NFS - OSI Model
8

9. RPC
Client Server
Process starts
Server waits
RPC Message
Server start
Client waits
PC
Server waits PE
PR
RPC Return
Process continued Termination
9

10. HDFS Layer
Local Client
HDFS Application
POSIX API HDFS API
VFS NFS Driver Network HDFS
10

11. HDFS Model
Client
add Blck (src)
HDFS Cluster
Namenode
write
DN Block received
Pipeline
DN Block received
DN Block received
10

12. HDFS Write Model
RPC (ClientProtocol) RPC
Client DFS NN
rcv only
FSData stream (socket)
RPC (DFSClient.DFSInputStream)
RPC proxy
DN intern
DN RPC
(DataNodeProtocol) RPC
xceiver IPC
Proxy
VFS
RPC proxy
RPC HDD
DN (DataNodeProtocol)
11

13. Links / Resources
The Hadoop Distributed File System
Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler,Yahoo!
http://dtrace.org/blogs/brendan/2011/05/11/
NFS and RPC
Chavalit Srisathapornphat, CISC856
Linux I/O Schedulers
Hao-Ran Liu
13