Add and Remove GlusterFS Servers

This article describes how to add and remove GlusterFS servers from your server array.

Create a new server

Use the nova boot command from the previous article to create a server called web3:

nova boot --image bb02b1a3-bc77-4d17-ab5b-421d89850fca --flavor performance1-4 web3

You can also use the Rackspace Cloud Control Panel to create the new server.

Add the server to the Rackspace custom network

In the previous article, you added a Rackspace custom network.

Get the Universal Unique Identifer (UUID) of the network by using the following nova command:

nova network-list
After you have the UUID, associate the new host with it.

Replace UUID in the following command with the actual UUID (for example, 5492de89-1497-4aa0-96eb-bcdd55e1195c1). web03 is the host name of the server that you want to add.
```
nova network-associate-host UUID web03
```

You can also use the Rackspace Cloud Control panel to associate a server with your existing network.

When you are done, the new server should have the IP address 192.168.0.3 on interface /dev/eth3. That is the address that GlusterFS uses to communicate with the other server.

Format the partition and install GlusterFS

Use the following steps to format the server partition and install GlusterFS:

Use Secure Shell (SSH) to log in to the server.

By using the instructions from the previous article, install GlusterFS and format the bricks partition as follows:

apt-get update
apt-get install -y glusterfs-server glusterfs-client
apt-get install -y xfsprogs
mkfs.xfs /dev/xvde1
mkdir /srv/.bricks
echo /dev/xvde1 /srv/.bricks xfs rw 0 1 >> /etc/fstab
mount /srv/.bricks
ufw allow in on eth2

Incorporate the new brick into the Gluster volume

Use SSH to log in to either web01 or web02.
The following command instructs the GlusterFS volume to trust the new server:
```
root@web02 :~# gluster peer probe 192.168.0.3
peer probe: success
```

Add the brick into the volume by using the following commands:

root@web02 :~# gluster volume add-brick www replica 3 192.168.0.3:/srv/.bricks/www
volume add-brick: success

The command includes the following parts:

gluster - The command is for GlusterFS.
volume - The command is related to a volume.
add-brick - You are adding a brick to the volume.
www - This is the name of the volume.
replica 3 - After you add this brick, the volume keeps at least three copies of each file, one copy per brick, and in this case, one copy per server (because only one brick is on each server).
192.168.0.3:/srv/.bricks/www - This part of the command is the IP address of the Gluster server, followed by the absolute path to where the brick data is stored.

Volume storage strategies

GlusterFS offers the following types of volume storage strategies:

Distributed - One file is on one brick, and the next file is on the next brick. This strategy gives you more space because your volume is the sum of all the bricks.
Replicated (Recommended) - Every file is copied to every server.
Striped - Files are cut into chunks, and one chunk is written to the first brick, one chunk is written to the second brick, and so on.

You can also combine strategies, for example replicated and distributed, as illustrated in the following example:

gluster volume create www replica 2 transport tcp
192.168.0.1:/srv/.bricks/www 192.168.0.2:/srv/.bricks/www
192.168.0.3:/srv/.bricks/www 192.168.0.4:/srv/.bricks/www

The replica number is the number of bricks that make up a replica set, that is, hold a full copy of the files. In the preceding example 192.168.0.1 and 192.168.0.2 hold a full copy of the files, as do 192.168.0.3 and 192.168.0.4. The brick order is significant. If you ordered it 1,3,2,4, then 1 and 3 and 2 and 4 would hold the full files. If 1 and 2 went down, you would lose half of your files and have two copies of the other half.

Having a combined of a replicated and distributed volume gives you a little extra speed, and more space in exchange for data safety.

A combination of striped and replicated volumes are recommended only when you have files that are bigger than a brick, or many large files that are undergoing many IO operations.

View the state of the servers

The following commands help you find out more about what's happening in your cluster. You will use these commands in later GlusterFS articles.

peer status

If you run the following command from any server, it shows all the peer servers that it knows about:

root@web01:~# gluster peer status
Number of Peers: 2

Hostname: 192.168.0.3
Uuid: ba502dc2-447f-466a-a732-df989e71b551
State: Peer in Cluster (Connected)

Hostname: 192.168.0.2
Uuid: 56e02356-d2c3-4787-ae25-6b46e867751a
State: Peer in Cluster (Connected)

volume status

The following command is a helpful troubleshooting command. It gives information about all the GlusterFS volumes and tasks queued and in progress.

root@web03:~# gluster volume status
Status of volume: www
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick 192.168.0.2:/srv/.bricks/www			49152	Y	13673
Brick 192.168.0.1:/srv/.bricks/www			49152	Y	10249
Brick 192.168.0.3:/srv/.bricks/www			49153	Y	13783
NFS Server on localhost					2049	Y	13793
Self-heal Daemon on localhost				N/A	Y	13800
NFS Server on 192.168.0.2				2049	Y	13900
Self-heal Daemon on 192.168.0.2				N/A	Y	13907
NFS Server on 192.168.0.1				2049	Y	10286
Self-heal Daemon on 192.168.0.1				N/A	Y	10293

There are no active volume tasks

Remove a brick

You can shrink volumes as needed while the cluster is online and available. Use the following command to remove a brick:

# gluster volume remove-brick <volName> <brickName> start

Running remove-brick with the start option automatically triggers a rebalance operation to migrate data from the removed bricks to the rest of the volume.

The following example removes brick 2:

root@web01:~# gluster volume remove-brick www replica 2 192.168.0.2:/srv/.bricks/www start
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success

This command tells GlusterFS that the www volume will now keep only 2 copies of each file. It warns you that you might lose data and prompts you to continue.

If you want to remove a brick on a distributed volume, you should to run the following command instead:

root@web01:~# gluster volume remove-brick www 192.168.0.2:/srv/.bricks/www start

You can view the status of the remove-brick operation by using the following command:

root@web01:~# watch gluster volume remove-brick www replica 2 192.168.0.2:/srv/.bricks/www status

Re-add a brick

This section explains how to re-add a brick.

Try adding web02 back into the volume, as follows:

root@web02:~# gluster volume add-brick www replica 3 192.168.0.2:/srv/.bricks/www
volume add-brick: failed:

It failed. Look at the logs on web02 to see why the command failed:

root@web02:/srv/.bricks# tail /var/log/glusterfs/*log -f | grep E
[2014-05-25 00:19:04.954410] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2014-05-25 00:19:12.958620] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2014-05-25 00:40:46.923747] E [glusterd-utils.c:5377:glusterd_is_path_in_use] 0-management: /srv/.bricks/www or a prefix of it is already part of a volume
[2014-05-25 00:40:46.923789] E [glusterd-op-sm.c:3719:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Add brick', Status : -1

The issue is that /srv/.bricks/www still contains the data from the time when web02 was a member of the volume.

You need a clean place to store the data. The easiest way to clean up is to just remove all the data by using the following command:

root@web02:~# rm -rf /srv/.bricks/www

🚧
Be careful to perform this action on the correct host (web02, which is currently out of the volume). If you make a mistake, the next article shows you how to recover. Alternative actions are to move the www directory out of the way, or to add the brick using another directory, such as www2.

Now that you have a clean location in which to store the brick, adding the brick is successful:

root@web01:/srv# gluster volume add-brick www replica 3 192.168.0.2:/srv/.bricks/www
volume add-brick: success

Next steps

How to recover from a failed server in a GlusterFS array

Create a new server

Add the server to the Rackspace custom network

Format the partition and install GlusterFS

Incorporate the new brick into the Gluster volume

Volume storage strategies

View the state of the servers

peer status

volume status

Remove a brick

Re-add a brick

🚧Be careful to perform this action on the correct host (web02, which is currently out of the volume). If you make a mistake, the next article shows you how to recover. Alternative actions are to move the www directory out of the way, or to add the brick using another directory, such as www2.

Next steps