https://docs.google.com/a/expertus.com/spreadsheets/d/1ZS4962TeT67_BPJw4yIbbmlFt_IOqvFlmLednp8b1ck/edit#gid=0

Step 1 - stop services and dismount volumes on prdapp01

(a) ssh expprodl@10.166.152.11

(b) stop god and apache on prdapp01 - 10.166.152.11

 /etc/init.d/god stop (As user expprodl)
 ~/bin/apache_sss.sh stop (As user expprodl)

(C) su root

(d) umount the following mount points

  umount /deploy/crossbow/shared/assets
  umount /deploy/crossbow/shared/system/uploaded_datas
  umount /deploy/crossbow/shared/system/images
  umount /deploy/crossbow/shared/system/images
  umount /deploy/crossbow/shared/system/sheets
  umount /deploy/crossbow/shared/system/site_backgrounds
  umount /deploy/crossbow/shared/system/site_logos
  umount /deploy/crossbow/shared/system/styles
  umount /deploy/crossbow/shared/system/templates
  umount /deploy/crossbow/shared/streams

Step (2) - stop services and dismount volumes on prdapp02

(a) ssh expprodl@10.166.152.19

(b) stop god and apache on prdapp02 - 10.166.152.19

 /etc/init.d/god stop (As user expprodl)
 ~/bin/apache_sss.sh stop (As user expprodl)

© su root

(d) umount below volumes

  umount /deploy/crossbow/shared/assets
  umount /deploy/crossbow/shared/system/uploaded_datas
  umount /deploy/crossbow/shared/system/images
  umount /deploy/crossbow/shared/system/images
  umount /deploy/crossbow/shared/system/sheets
  umount /deploy/crossbow/shared/system/site_backgrounds
  umount /deploy/crossbow/shared/system/site_logos
  umount /deploy/crossbow/shared/system/styles
  umount /deploy/crossbow/shared/system/templates
  umount /deploy/crossbow/shared/streams

Verify that the maintenance page is up.

Step (3)- upgrade GlusterFS server from 3.2.4 to 3.5 in db slave box

(a) ssh expprodl@10.166.152.13

(b) su root

© /etc/init.d/glusterfs stop

  stop glusterd
  # nfs processes etc.
  sudo kill `pgrep gluster`
  sudo portmap stop

(d) backup the following mount points

  cd /data00/
  tar -pcvzf /backup_glusterfs/assets_20140728.tar.gz assets/
  tar -pcvzf /backup_glusterfs/images_20140728.tar.gz images/
  tar -pcvzf /backup_glusterfs/sheets_20140728.tar.gz sheets/
  tar -pcvzf /backup_glusterfs/site_backgrounds_20140728.tar.gz site_backgrounds/
  tar -pcvzf /backup_glusterfs/site_logos_20140728.tar.gz site_logos/
  tar -pcvzf /backup_glusterfs/styles_20140728.tar.gz styles/
  tar -pcvzf /backup_glusterfs/templates_20140728.tar.gz templates/
  tar -pcvzf /backups_glusterupg/sharedpub_20140728.tar.gz sharedpub/
  
 

(e) cd /etc

  tar cvzf glusterd_26_Jul_2014.tar.gz glusterd
  tar cvzf glusterfs_26_Jul_2014.tar.gz glusterfs (this does not seem to be a valid configuration folder)

(f) upgrade glusterfs on dbslave

step f1 compile - install gluster - did not work

  mkdir /opt/src/glusterfs
  cd /opt/src/glusterfs
  wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/glusterfs-3.5.0.tar.gz
  tar zxvf glusterfs-3.5.0.tar.gz
  cd glusterfs-3.5.0
  ./autogen.sh #might prompt to install dependencies
  make and make install
  
  Used the rpms made by gluster.org
  http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.0/CentOS/epel-5Server/x86_64/glusterfs-libs-3.5.0-2.el5.x86_64.rpm
  http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.0/CentOS/epel-5Server/x86_64/glusterfs-fuse-3.5.0-2.el5.x86_64.rpm
  http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.0/CentOS/epel-5Server/x86_64/glusterfs-server-3.5.0-2.el5.x86_64.rpm
  http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.0/CentOS/epel-5Server/x86_64/glusterfs-3.5.0-2.el5.x86_64.rpm

step f2

setfattr -n trusted.glusterfs.volume-id -v 0x5e957f5d47c64509b0651e0c1123d286 /data00/assets
setfattr -n trusted.glusterfs.volume-id -v 0xe1015abf7ce34697ab0b94b9a1558d7e /data00/sharedpub
setfattr -n trusted.glusterfs.volume-id -v 0x233515755b144eee8d95621199d121bb /data00/images
setfattr -n trusted.glusterfs.volume-id -v 0x7fba152b45b5458d8d002fab012257a7 /data00/sheets
setfattr -n trusted.glusterfs.volume-id -v 0xd7c0e1d7581e4449a1ce00ec2a8f336c /data00/site_backgrounds
setfattr -n trusted.glusterfs.volume-id -v 0x4b1017c9d7d5400282f5453e6fb2045c /data00/site_logos
setfattr -n trusted.glusterfs.volume-id -v 0xd31fdc7fcc7f4f1c9964f392f5f94846 /data00/styles
setfattr -n trusted.glusterfs.volume-id -v [volid]                            /data00/templates

step f3

gluster volume status
killall glusterd
glusterd --xlator-option *.upgrade=on -N (did not this step for RPM upgrade as in gluster.org notes /Vijay Bellur)
start glusterd

Step 4 - upgrade glusterfs client on prdapp02 - 10.166.152.19

cd /etc/yum.repos.d/
wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/CentOS/glusterfs-epel.repo.el5
mv glusterfs-epel.repo.el5 glusterfs-epel.repo
yum install glusterfs-client

test glusterfs connectivity - app server02 to db slave
mkdir /tmp/a
mount.glusterfs 10.166.152.13:/site_logos /tmp/a 
umount /tmp/a

Step 5 - upgrade glusterfs client on prdapp01 - 10.166.152.11

ssh expprodl@10.166.152.11
su root
cd /etc/yum.repos.d/
wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/CentOS/glusterfs-epel.repo.el5
mv glusterfs-epel.repo.el5 glusterfs-epel.repo
yum install glusterfs-client

test glusterfs connectivity - app server01 to db slave
mkdir /tmp/a
mount.glusterfs 10.166.152.13:/site_logos /tmp/a
umount /tmp/a

step 6 - upgrade glusterfs server on prdcollab01 (ubuntu) - 10.166.152.17

step 6.1 - stop BBB on prdcollab01 - 10.166.152.17

        ssh expbbbp@10.166.152.17
        sudo su -
        /etc/init.d/glusterfs stop

step 6.2 - upgrade glusterfs server on prdcollab01 - 10.166.152.17

       Stop all glusterd processes.
         stop glusterd
         # nfs processes etc.
         sudo kill `pgrep gluster`
         sudo portmap stop

        Take a backup of /etc/glusterd.

        backup -/usr/share/red5/webapps/video/streams

        Add PPA and Install Gluster 3.5
           add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5
           apt-get install gluster-server gluster-client
     

        The installation process will move /etc/glusterd/* to /var/lib/glusterd/*. 
        Verify directory structure. New files might be added, thats OK.
        If the previous step fails (i.e. files were not migrated), do a manual copy.
        
        mkdir -p /var/lib/glusterd
        
         cp -r /etc/glusterd/* /var/lib/glusterd/
         
         Start glusterd in upgrade mode. 
         It will upgrade existing volumes' index files to support new version.
         
         killall glusterd
         glusterd --xlator-option *.upgrade=on -N
         start glusterd

setfattr -n truster.glusterfs.volume-id -v 0x (info file has the file attr)

symlink /usr/share/red5/streams to /opt/streams

step 7 Reboot prddbslav01
step 8 Reboot prdcollab01
step 9 verify Gluster server services
step 10 check ps -ef | grep gluster on all the boxes and compare it against the output already saved

step 11 Reboot prdapp01
step 12 Reboot prdapp02

Step 13 - Mount all endpoints on app1 and app2 (if they are not already mounted after the reboot)

 mount.glusterfs 10-166-152-13:/asset-volume     /deploy/crossbow/shared/assets
 mount.glusterfs 10-166-152-13:/data-volume      /deploy/crossbow/shared/system/uploaded_datas
 mount.glusterfs 10-166-152-13:/images-volume    /deploy/crossbow/shared/system/images
 mount.glusterfs 10-166-152-13:/site_backgrounds /deploy/crossbow/shared/system/site_backgrounds
 mount.glusterfs 10-166-152-13:/site_logos       /deploy/crossbow/shared/system/site_logos
 mount.glusterfs 10-166-152-13:/styles           /deploy/crossbow/shared/system/styles
 mount.glusterfs 10-166-152-13:/sheets           /deploy/crossbow/shared/system/sheets
 mount.glusterfs 10-166-152-13:/templates        /deploy/crossbow/shared/system/templates
 mount.glusterfs prodcollab01.learninguniver.se:/streams  /deploy/crossbow/shared/streams

Set the file permissions appropriately at the top level of the mount on prdapp01 / prdapp02

Step 14 test (a) Put app02 in LB maintanance mode

  ssh root@10.166.152.16
  echo "enable server learnexa/prodapp02" | socat stdio /var/run/haproxy/haproxy.sock
  

(b) Bring app02 service up and test.

/etc/init.d/god stop (As user expprodl)

~/bin/apache_sss.sh stop (As user expprodl)

IF app02 testing is fine , Test app01 like (a) and (b)

IF both app01 and app02 are fine go to 15a

ELSE go to 15b

Step 15a - Bring the site up

ssh root@10.166.152.16

  echo "enable server learnexa/prodapp01" | socat stdio /var/run/haproxy/haproxy.sock
  echo "enable server learnexa/prodapp02" | socat stdio /var/run/haproxy/haproxy.sock
  

Step 15b - Rollback

Rollback - Glusterfs server/client on prddb-slave(centos)
Uninstall glusterserver 3.5.0/1
Uninstall glusterclient 3.5.0/1
install glusterserver 3.2.4 (rpm for core and fuse are in dbslave - /opt/src/glusterfs)
install glusterclient 3.2.4
restore backed up /etc/gluster
restore backed up /data00/folders
setfattr if needed (but do not think it is necessary)
reboot
verify glusterfs services are started

Rollback - Glusterfs server on prdcollab01(ubuntu)

Uninstall glusterserver 3.5.0/1
Uninstall glusterclient 3.5.0/1
install glusterserver 3.2.4(ubuntu version / source needs to be searched out)
install glusterclient 3.2.4(ubuntu version / source needs to be searched out)
restore backed up /etc/gluster
restore backed up /streams mount
setfattr if needed (but do not think it is necessary)
reboot
verify glusterfs services are started

Rollback - Glusterfs client on prdapp01(centos)

uninstall glusterclient 3.5.0/1
install glusterclient 3.2.4 (centos rpm available in prdb-slave /opt/glusterfs)
reboot
mount all the volumes (not necessary to do it manually - mtab should take care - but mount commands with mount points are documented above)

Rollback - Glusterfs client on prdapp02(centos)

uninstall glusterclient 3.5.0/1
install glusterclient 3.2.4
reboot
mount all the volumes (not necessary to do it manually - mtab should take care - but mount commands with mount points are documented above)