DokuWiki

Production GlusterFs Upgrade

Ship list - https://docs.google.com/a/expertus.com/spreadsheets/d/1ZS4962TeT67_BPJw4yIbbmlFt_IOqvFlmLednp8b1ck/edit#gid=0

1. Two glusterFS servers, one for shared directories hosted on DB Slave and the other for streaming files hosted on BBB server.
   
2. Bigger shares.
   
3. Hybrid centos and ubuntu host systems.
   
4. Clients (app servers) consume from both the servers hence it is necessary to upgrade both the GlusterFS server versions so the client version can be in sync.
   

2 - 2.30 hours.

0) Bring app servers down.
—MAINT— 1) Stop God and Apache on both app servers.
2) Stop BBB.
3) Backup shared directory on DB Slave.
4) Upgrade GlusterFS Server on DB Slave (steps below, after this section)
5) Upgrade GlusterFS Client on App server 2.
6) Test GlusterFS connectivity - app server 2 to DB Slave.

mkdir /tmp/a
mount.glusterfs 10.166.152.13:/site_logos /tmp/a 

* At this point, we have covered any potentially unknown issues *
7) Upgrade GlusterFS Client on App Server 1.
8) Upgrade GlusterFS Server on BBB and start BBB.
9) Mount all endpoints.
10) Test.
11) Bring the site up (incl. starting god and apache).

GlusterFS 3.5 is not compatible with current version in production i.e. 3.2.4.

1. Downtime will be necessary across all environments for the upgrade procedure.
2. Stop apache, god and bbb.
3. Unmount any mounted gluster filesystems.

mount
# make a note of filesystems
umount x

1. Stop all glusterd processes.

stop glusterd
# nfs processes etc.
sudo kill `pgrep gluster`
sudo portmap stop

1. Take a backup of /etc/glusterd.
2. Go the extra mile: take a backup of the share directories
3. Add PPA and Install Gluster 3.5

    add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5
    apt-get install gluster-server gluster-client

1. The installation process will move /etc/glusterd/* to /var/lib/glusterd/*.
   Verify directory structure. New files might be added, thats OK.
2. If the previous step fails (i.e. files were not migrated), do a manual copy.

mkdir -p /var/lib/glusterd
cp -r /etc/glusterd/* /var/lib/glusterd/

1. Start glusterd in upgrade mode.
   It will upgrade existing volumes' index files to support new version.

killall glusterd
glusterd --xlator-option *.upgrade=on -N
start glusterd

1. Build from yum

cd /etc/yum.repos.d/
wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/CentOS/glusterfs-epel.repo.el5
mv glusterfs-epel.repo.el5 glusterfs-epel.repo
yum install glusterfs-client

1. Ensure that Step 1 and Step 2 from Server section above have been performed.
2. Build from Source

mkdir /opt/src/glusterfs
cd /opt/src/glusterfs
wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/glusterfs-3.5.0.tar.gz
tar zxvf glusterfs-3.5.0.tar.gz
cd glusterfs-3.5.0
./autogen.sh #might prompt to install dependencies
make and make install

- Set attr for each brick

vol=<VOLNAME>
brick=<BRICK_PHSYICAL_PATH> # ex: /usr/share/red5/streams
setfattr -n trusted.glusterfs.volume-id \
  -v 0x$(grep volume-id /var/lib/glusterd/vols/$vol/info \
  | cut -d= -f2 | sed 's/-//g') $brick

1. Verify if volumes can be mounted.
2. Verify it Gluster was able to bring the volumes up.

gluster volume status

1. Ensure Gluster Server comes up post reboot.
2. Start all services and bring servers up.

Identifying an exact point where it stops can help.
Append –debug to a mount command.
Example:

mount.glusterfs 10.166.152.13:/styles /mnt/styles

• Freezes at startup.

gluster peer status

If it reports “Peer Rejected (Connected)”, execute on the GLUSTER SERVER

gluster peer detach <IPADDRESS_OF_CLIENT>

and then

gluster peer probe <IPADDRESS_OF_CLIENT>

If it still freezes, continue with the removal of metadata.

[2014-02-11 21:49:07.293875] D [glusterd-store.c:2109:glusterd_store_peer_write] 0-: Returning with 0
[2014-02-11 21:49:07.293921] D [glusterd-store.c:2137:glusterd_store_perform_peer_store] 0-: Returning 0
[2014-02-11 21:49:07.293952] D [glusterd-store.c:2158:glusterd_store_peerinfo] 0-: Returning with 0
[2014-02-11 21:49:07.294334] I [glusterd-rpc-ops.c:715:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 

Fix: The directory will be recreated with the next successful sync. Take a backup, if needed in between.

rm /etc/glusterd -rf

• Network related
  Gluster listens on port 24007. netstat, lsof -i .

ls /mounted/dir
#replace with the actual path

Fix: Note mounted directories via gluster

mount
#keep a record of the mounted directories
#umount them, one by one 
umount <directory noted earlier>

sudo kill `pgrep gluster`
sudo portmap restart
sudo glusterd
mount -a

Test:

ls /mounted/dir
#replace with the actual path

Notes:

1. This issue is only observed when glusterd 3.2.4 server is running on Ubuntu server.
2. It is known that it does not affect glusterd 3.2.4 server running on CentOS.