Gluster 3.1 introduces important new features like the Gluster Console Manager command line tool for simplifying storage configuration and management. It allows dynamic management of GlusterFS volumes without restarting servers or requiring clients to reload volumes. Support for NFSv3 is also built-in. The document discusses how to install, configure, and manage GlusterFS volumes including creating volume types, adding or removing bricks, rebalancing, and migrating data.

1. Gluster 3.1介绍与使用
DP

2. 议题
 新增重要功能
 编译安装
 创建存储群集
 三种卷（Volume）类型
 创建配置GlusterFS卷
 管理GlusterFS卷：扩容/缩减迁移重平衡
 卷的使用过程
 日志管理
 访问GlusterFS
DP

3. Gluster 3.1新增重要功能
控制台
GlusterFS 3.1中提供了一个叫Gluster Console Manager的命令行工具用于简化存储环境的配
置和管理。Gluster Console Manager提供的功能类似于LVM CLI(Logical Volume Manager)和ZFS
Command Line Interface，但跨多个存储服务器。你可以在卷加载和活动时，使用Gluster
Console Manager。
卷的动态管理
可以动态更改GlusgerFS的卷配置而不需要重启服务器或也不需要客户端重新加载卷。你可
以使用一个新的守护进程glusterd来实现这种动态卷的管理。使用glusterd命令，你可以将卷
轻易的扩展、收缩、迁移和重新均衡而不需要计划停机时间。
NFS支持
内建对NFSv3协议的支持。
DP

4. Gluster控制台
GlusterFS 3.1中提供了一个叫Gluster Console Manager的命令行工具用于简化存储环境的配置和管理。
[root@glusterc01 glusterfs]# gluster
gluster> help
volume info [all|<VOLNAME>] - list information of all volumes
volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [transport <tcp|rdma>] <NEW-BRICK> ...
- create a new volume of specified type with mentioned bricks
volume delete <VOLNAME> - delete volume specified by <VOLNAME>
volume start <VOLNAME> [force] - start volume specified by <VOLNAME>
volume stop <VOLNAME> [force] - stop volume specified by <VOLNAME>
volume add-brick <VOLNAME> [(replica <COUNT>)|(stripe <COUNT>)] <NEW-BRICK> ... - add brick to volume
<VOLNAME>
volume remove-brick <VOLNAME> [(replica <COUNT>)|(stripe <COUNT>)] <BRICK> ... - remove brick from
volume <VOLNAME>
volume rebalance <VOLNAME> start - start rebalance of volume <VOLNAME>
volume rebalance <VOLNAME> stop - stop rebalance of volume <VOLNAME>
volume rebalance <VOLNAME> status - rebalance status of volume <VOLNAME>
volume replace-brick <VOLNAME> (<BRICK> <NEW-BRICK>) start|pause|abort|status - replace-brick
operations
volume set <VOLNAME> <KEY> <VALUE> - set options for volume <VOLNAME>
volume help - display help for the volume command
volume log filename <VOLNAME> [BRICK] <PATH> - set the log file for corresponding volume/brick
volume log locate <VOLNAME> [BRICK] - locate the log file for corresponding volume/brick
volume log rotate <VOLNAME> [BRICK] - rotate the log file for corresponding volume/brick
volume sync <HOSTNAME> [all|<VOLNAME>] - sync the volume information from a peer
volume reset <VOLNAME> - reset all the reconfigured options
peer probe <HOSTNAME> - probe peer specified by <HOSTNAME>
peer detach <HOSTNAME> - detach peer specified by <HOSTNAME>
peer status - list status of peers
peer help - Help command for peer
quit - quit
help - display command options
gluster>
DP

5. 动态管理
1 显示存储卷的信息 4 重复添加多个存储单元
#gluster volume info esign
Volume Name: esign 5 显示变更后存储卷的信息
Type: Replicate
Status: Started #gluster volume info esign
Number of Bricks: 2 Volume Name: esign
Transport-type: tcp Type: Replicate
Bricks: Status: Started
Brick1: server1:/export1 Number of Bricks: 8
Brick2: server2:/export1 Transport-type: tcp
2 增加两台服务器 使用命令代替原有配 Bricks:
Brick1: server1:/export1
Brick2: server2:/export1
置档案
# gluster peer probe server3
Brick3: server3:/export1
# gluster peer probe server4
Brick4: server4:/export1
Brick5: server1:/export2
3 添加两个存储单元（Brick）
不需要重启服务
# gluster volume add-brick esign server3:/export
# gluster volume add-brick esign server4:/export
Brick6: server2:/export2
Brick7: server3:/export2
Brick8: server4:/export2
BRICK1 BRICK 2 BRICK 3 BRICK 4
BRICK5 BRICK 6 BRICK 7 BRICK 8
SERVER1 SERVER2 SERVER3 SERVER4
DP

6. 编译&安装GlusterFS
1. 建立新的目录
# mkdir glusterfs
# cd glusterfs
2.下载源码
源码下载地址：http://www.gluster.org/download/
3. 将源代码包解压
# tar -xvf <source file>
4. 执行配置命令
# ./configure
GlusterFS configure summary
==================
FUSE client : yes
Infiniband verbs : yes
epoll IO multiplex : yes
argp-standalone : no
fusermount : no
readline : yes
5. 编译安装
# make
# make install
6. 确认安装GlusterFS的版本
# glusterfs --version
DP

7. 启动GlusterFS
1. 手动启动
启动
# /etc/init.d/glusterd start
停止
# /etc/init.d/glusterd stop
2. 设置自动启动
# chkconfig glusterd on
DP

8. 创建存储群集
1. 向资源池中添加服务器
# gluster peer probe SERVER
例如共有四台存储服务器，在server1上添加其它三台服务器：
# gluster peer probe server2
Probe successful
# gluster peer probe server3
Probe successful
# gluster peer probe server4
Probe successful
注意: 不可以添加自己 (server 1)。
2.确认各节点状态
#gluster peer status
Number of Peers: 3
Hostname: 192.168.1.92
Uuid: 5e987bda-16dd-43c2-835b-08b7d55e94e5
State: Peer in Cluster (Connected)
Hostname: 192.168.1.93
Uuid: 1e0ca3aa-9ef7-4f66-8f15-cbc348f29ff7
State: Peer in Cluster (Connected)
Hostname: 192.168.1.94
Uuid: 345de43e-4533-4e33-4f77-ed3984da21ae
State: Peer in Cluster (Connected)
DP

9. 三种卷（Volume）类型
分布式 复制卷 条带卷
类型
(Distributed Volumes ) （Replicated Volume ) (Striped Volume )
图例
BRICK1 BRICK2 BRICK1 BRICK2 BRICK1 BRICK2
SERVER 1 SERVER 2 SERVER 1 SERVER 2 SERVER 1 SERVER 2
文件被分散存储在 文件被存储多份 文件被分割后存储
说明
存储单元 在各存储单元里
优势
DP

10. 创建配置GlusterFS卷
分布式卷(Distributed Volumes )
# gluster volume create NEW-VOLNAME [transport tcp | rdma] NEW-BRICK...
例如创建一个由四台服务器组成的分布式卷
# gluster volume create test-volume transport tcp server1:/exp1 server2:/exp2 server3:/exp3
server4:/exp4
Creation of volume distribute has been successful
复制卷(Replicated Volume )
# gluster volume create NEW-VOLNAME [replica COUNT] [transport tcp | rdma] NEW-BRICK...
例如创建一个由两台服务器组成的复制卷
# gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2
Creation of volume replicate has been successful
注意：存储单元（Brick）的数量必须为复制数量的整数倍。
条带卷(Striped Volume )
# gluster volume create NEW-VOLNAME [stripe COUNT] [transport tcp | rdma] NEW-BRICK...
例如创建一个由两台服务器组成的复制卷
# gluster volume create test-volume stripe 2 transport tcp server1:/exp1 server2:/exp2
Creation of volume replicate has been successful
注意：存储单元（Brick）的数量必须为条带数的整数倍。
新建的卷需要启动后才能使用
# gluster volume start VOLNAME
# gluster volume start test-volume
详见：http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Configuring_GlusterFS_Volumes
Starting volume test-volume has been successful DP

11. 管理GlusterFS卷（一）扩容/缩减
卷的扩容(Expanding Volumes) 卷的缩减(Shrinking Volumes)
1 在群集中的添加新的服务器。 1 删除存储单元（Brick）。
# gluster peer probe HOSTNAME # gluster volume remove-brick VOLNAME BRICK
# gluster peer probe server4 # gluster volume remove-brick test-volume
Probe successful server2:/exp2
Removing brick(s) can result in data loss.
2 添加存储单元
Do you want to Continue? (y/n)
# gluster volume add-brick VOLNAME NEW-BRICK 2 输入y，确认继续执行，将显示下面的信息：
For example, to add server4:/exp4 to test-volume:
Remove Brick successful
# gluster volume add-brick test-volume
server4:/exp4 3 检查卷的信息
Add Brick successful
# gluster volume info
3 检查卷的信息 Volume Name: test-volume
Type: Distribute
# gluster volume info Status: Started
Volume Name: test-volume Number of Bricks: 3
Type: Distribute Bricks:
Status: Started Brick1: server1:/exp1
Number of Bricks: 4 Brick3: server3:/exp3
Bricks: Brick4: server4:/exp4
Brick1: server1:/exp1
Brick2: server2:/exp2 4 重新平衡群集中数据
Brick3: server3:/exp3
Brick4: server4:/exp4
4 重新平衡群集中数据
提示： 当要对复制卷和条带卷扩容或缩减时，添加或删除的存储单元（Brick）数量必须是复制数或条带数的倍数。例如，要对一
个复制数为2的卷扩容，你添加的Brick数量应为2的倍数，如4、6、8等。
DP

12. 管理GlusterFS卷（二）迁移
卷的迁移(Migrating Volumes)
1 将数据从一个Brick迁移到另外一个Brick，执行：
# gluster volume replace-brick VOLNAME BRICK NEW-BRICK start
例如：将test-volume卷中server3:/exp3的数据迁移到server5:/exp5
# gluster volume replace-brick test-volume server3:/exp3 server5:exp5 start
Replace brick start operation successful
注释：在执行replace-brick命令前，确保服务器上已安装FUSE软件包
2 暂停迁移
# gluster volume replace-brick VOLNAME BRICK NEW-BRICK pause
例如：暂停从server3:/exp3到server5:/exp5的数据迁移
# gluster volume replace-brick test-volume server3:/exp3 server5:exp5 pause
Replace brick pause operation successful
3 终止迁移
# gluster volume replace-brick VOLNAME BRICK NEW-BRICK abort
例如：终止从server3:/exp3到server5:/exp5的数据迁移
# gluster volume replace-brick test-volume server3:/exp3 server5:exp5 abort
Replace brick abort operation successful
4 查看迁移状态
# gluster volume replace-brick VOLNAME BRICK NEW-BRICK status
例如：查看从server3:/exp3到server5:/exp5的数据迁移状态
# gluster volume replace-brick test-volume server3:/exp3 server5:/exp5 status
Current File = /usr/src/linux-headers-2.6.31-14/block/Makefile Number of files
migrated = 10567
显示当前迁移的文件和已迁移文件的数量。 DP

13. 管理GlusterFS卷（三）重平衡
注释： 在完成对卷的扩容或缩减后，需要重新平衡存储各服务器的数据。
卷的重平衡(Rebalancing Volumes )
1 执行重平衡
# gluster volume rebalance VOLNAME start
# gluster volume rebalance test-volume start
Starting defrag on volume test-volume has been successful
2 检查当前状态
step 1: layout fix in progress: fixed layout 36
# gluster volume rebalance VOLNAME status
# gluster volume rebalance test-volume status
Rebalance in progress: rebalanced 399 files of size 302047 (total files scanned 765)
完成时间取决于文件的数量及其大小。检查当前状态可以了解当前已经完成的文件数量和已扫描的文件数量。
当重平衡完成后，状态显示如下：
# gluster volume rebalance test-volume status
Rebalance completed!
3 暂停重平衡
# gluster volume rebalance VOLNAME stop
# gluster volume rebalance test-volume stop
Stopping defrag on volume test-volume has been successful
重新开始将继续执行。
DP

14. 卷的使用过程
创建新的卷 修改卷的选项
# gluster volume create NEW-VOLNAME [stripe COUNT]
# gluster volume set VOLNAME OPTION PARAMETER
[replica COUNT] [transport <tcp | rdma>] NEW-BRICK ...
默认传输类型为tcp。 # gluster volume set test-volume performance.cache-size 256MB
例如创建一个包括server3:/exp3和server4:/exp4名为test- Set volume successful
volume的卷。
# gluster volume create test-volume server3:/exp3 参数列表
server4:/exp4
1 performance.cache-min-file-size
performance.cache-max-file-size
( 0 < min-file-size )
( 0 < max-file-size > min-file-size )
启动卷 performance.cache-refresh-timeout ( 0 < cache-timeout < 61 )
# gluster volume start VOLNAME ( The subvolume name on which read operations should
# gluster volume start test-volume cluster.read-subvolume
be performed on)
2 cluster.stripe-block-size ( list such as : "*.jpg:1,abc*:2" )
显示卷的信息 cluster.self-heal-window-size ( 0 < data-self-heal-window-size < 1025)
# gluster volume info VOLNAME network.frame-timeout ( 5 < frame-timeout < 86400 )
# gluster volume info test-volume network.ping-timeout ( 0 < ping-timeout < 1014 )
Volume Name: test-volume auth.allow ( valid IP address which includes wild card
Type: Distribute
Status: Created
3 auth.reject patterns including *, such as 192.168.1.* )
performance.cache-size ( 4MB < cache-size < 6GB )
Number of Bricks: 4
Bricks: performance.write-behind-window-size ( Write-behind cache size )
Brick1: server1:/exp1 diagnostics.latency-measurement ( yes/no )
Brick2: server2:/exp2 diagnostics.dump-fd-stats ( yes/no )
Brick3: server3:/exp3 diagnostics.brick-log-level ( DEBUG|WARNING|ERROR|CRITICAL|NONE|TRACE )
Brick4: server4:/exp4
# gluster volume info all
diagnostics.client-log-level ( DEBUG|WARNING|ERROR|CRITICAL|NONE|TRACE )
4
停止卷 删除卷
a # gluster volume stop VOLNAME a # gluster volume delete VOLNAME
# gluster volume stop test-volume # gluster volume delete test-volume
Stopping volume will make its data inaccessible. Do Deleting volume will erase all information about
you want to continue? (y/n) 5b the volume. Do you want to continue? (y/n)
b 检查当前状态 检查当前状态
Stopping volume test-volume has been successful Stopping volume test-volume has been successful
DP

15. 日志管理
设置保存卷日志的路径
1 # gluster volume log filename VOLNAME DIRECTORY
例如将卷test-volume的日志保存在/var/log/test-volume/目录下。
# gluster volume log filename test-volume /var/log/test-volume/
log filename : successful
提示：在执行此命令前，确保目录已存在。
2 确认卷日志的存放路径
# gluster volume log locate VOLNAME
# gluster volume log locate test-volume
log file location: /var/log/test-volume
循环使用日志文件
# gluster volume log rotate VOLNAME
# gluster volume log rotate test-volume
log rotate successful
DP

16. 访问GlusterFS
使用NFS方式
#mount -t nfs HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR
# mount -t nfs server1:/test-volume /mnt/glusterfs
使用CIFS方式
在Windows中，点击“开始”->“运行”，然后输入：
SERVERNAMEVOLNAME
server1test-volume
使用Gluster客户端方式
# mount -t glusterfs HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR
# mount -t glusterfs server1:/test-volume /mnt/glusterfs
挂载参数：
log-level=loglevel
log-file=logfile
transport=transport-type
direct-io-mode=[enable|disable]
volume-name=volume_name
volume-id=volume-ID
DP

17. Troubleshooting
在使用过程中，曾碰到下面的错误
ERROR 1142 (0x00000476) Creating Destination Directory 目录名
An attempt was made to create more links on a file than the file system supports.
后来确认原来是ext3子目录数量限制（每个目录最多有31998子目录 ）所致，与
glusterfs并无关系。但要突破这一限制，需要将ext3分区升级为ext4的分区。
DP