=====Server Software=====

[[ Production GlusterFs Upgrade]]

==== Upgrade to Gluster 3.5 PROD ====

Ship list - https://docs.google.com/a/expertus.com/spreadsheets/d/1ZS4962TeT67_BPJw4yIbbmlFt_IOqvFlmLednp8b1ck/edit#gid=0

=== Differences between DEV / QA ===
  - Two glusterFS servers, one for shared directories hosted on DB Slave and the other for streaming files hosted on BBB server. \\ 
  - Bigger shares. \\ 
  - Hybrid centos and ubuntu host systems. \\ 
  - Clients (app servers) consume from both the servers hence it is necessary to upgrade both the GlusterFS server versions so the client version can be in sync. \\ 
=== Estimated downtime ===
2 - 2.30 hours. \\ 
=== Ship list ===
0) Bring app servers down.  \\ 
---MAINT---
1) Stop God and Apache on both app servers. \\ 
2) Stop BBB. \\ 
3) Backup shared directory on DB Slave. \\ 
4) Upgrade GlusterFS Server on DB Slave (steps below, after this section) \\ 
5) Upgrade GlusterFS Client on App server 2. \\ 
6) Test GlusterFS connectivity - app server 2 to DB Slave. \\ 
<code>
mkdir /tmp/a
mount.glusterfs 10.166.152.13:/site_logos /tmp/a 
</code>
 * At this point, we have covered any potentially unknown issues * \\ 
7) Upgrade GlusterFS Client on App Server 1. \\ 
8) Upgrade GlusterFS Server on BBB and start BBB. \\ 
9) Mount all endpoints. \\ 
10) Test. \\ 
11) Bring the site up (incl. starting god and apache). \\ 
==== Upgrade to Gluster 3.5====
===Server ===
GlusterFS 3.5 is not compatible with current version in production i.e. 3.2.4. \\
  - Downtime will be necessary across all environments for the upgrade  procedure.
  - Stop apache, god and bbb.
  - Unmount any mounted gluster filesystems.
<code>
mount
# make a note of filesystems
umount x
</code>
  - Stop all glusterd processes.
<code>
stop glusterd
# nfs processes etc.
sudo kill `pgrep gluster`
sudo portmap stop
</code>
  - Take a backup of /etc/glusterd.
  - Go the extra mile: take a backup of the share directories
  - Add PPA and Install Gluster 3.5
<code>
    add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5
    apt-get install gluster-server gluster-client
</code>
  - The installation process will move /etc/glusterd/* to /var/lib/glusterd/*. \\ Verify directory structure. New files might be added, thats OK.
  - If the previous step fails (i.e. files were not migrated), do a manual copy.
<code>
mkdir -p /var/lib/glusterd
cp -r /etc/glusterd/* /var/lib/glusterd/
</code>
  - Start glusterd in upgrade mode. \\ It will upgrade existing volumes' index files to support new version.
<code>
killall glusterd
glusterd --xlator-option *.upgrade=on -N
start glusterd
</code> 

=== CentOS ===
  - Build from yum
<code>
cd /etc/yum.repos.d/
wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/CentOS/glusterfs-epel.repo.el5
mv glusterfs-epel.repo.el5 glusterfs-epel.repo
yum install glusterfs-client
</code>
=== Ubuntu ===
  - Ensure that Step 1 and Step 2 from Server section above have been performed.
  - Build from Source
<code>
mkdir /opt/src/glusterfs
cd /opt/src/glusterfs
wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/glusterfs-3.5.0.tar.gz
tar zxvf glusterfs-3.5.0.tar.gz
cd glusterfs-3.5.0
./autogen.sh #might prompt to install dependencies
make and make install
</code>
 - Set attr for each brick
<code>
vol=<VOLNAME>
brick=<BRICK_PHSYICAL_PATH> # ex: /usr/share/red5/streams
setfattr -n trusted.glusterfs.volume-id \
  -v 0x$(grep volume-id /var/lib/glusterd/vols/$vol/info \
  | cut -d= -f2 | sed 's/-//g') $brick
</code>

  - Verify if volumes can be mounted.
  - Verify it Gluster was able to bring the volumes up.
<code>
gluster volume status
</code>
  - Ensure Gluster Server comes up post reboot.
  - Start all services and bring servers up.

==== Gluster 3.2.4 ====
=== Gluster does not mount external directories ===
== Diagnose and Fix ==
Identifying an exact point where it stops can help. \\ Append --debug to a mount command. \\
Example: 
<code>
mount.glusterfs 10.166.152.13:/styles /mnt/styles
</code>
  * Freezes at startup.
<code>
gluster peer status
</code>

If it reports "Peer Rejected (Connected)", execute on the GLUSTER SERVER
<code>
gluster peer detach <IPADDRESS_OF_CLIENT>
</code>
and then 
<code>
gluster peer probe <IPADDRESS_OF_CLIENT>
</code>

If it still freezes, continue with the removal of metadata.
<code>
[2014-02-11 21:49:07.293875] D [glusterd-store.c:2109:glusterd_store_peer_write] 0-: Returning with 0
[2014-02-11 21:49:07.293921] D [glusterd-store.c:2137:glusterd_store_perform_peer_store] 0-: Returning 0
[2014-02-11 21:49:07.293952] D [glusterd-store.c:2158:glusterd_store_peerinfo] 0-: Returning with 0
[2014-02-11 21:49:07.294334] I [glusterd-rpc-ops.c:715:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
</code>
   
Fix:
The directory will be recreated with the next successful sync. Take a backup, if needed in between.
<code>
rm /etc/glusterd -rf
</code>

  *   Network related \\ Gluster listens on port 24007. netstat, lsof -i .

=== Transport endpoint not connected ===
== Glusterd server version 3.2.4 on Ubuntu ==
== Glusterd client version 3.2.4 on CentOS / Ubuntu ==
== Diagnose and Fix ==
<code>
ls /mounted/dir
#replace with the actual path
</code>

Fix:
Note mounted directories via gluster
<code>
mount
#keep a record of the mounted directories
#umount them, one by one 
umount <directory noted earlier>

</code>
<code>
sudo kill `pgrep gluster`
sudo portmap restart
sudo glusterd
mount -a
</code>

Test:
<code>
ls /mounted/dir
#replace with the actual path
</code>

Notes:
  - This issue is only observed when glusterd 3.2.4 server is running on Ubuntu server.
  - It is known that it does not affect glusterd 3.2.4 server running on CentOS.