====== Production GlsuterFs Upgrade ====== ===== Ship List ===== https://docs.google.com/a/expertus.com/spreadsheets/d/1ZS4962TeT67_BPJw4yIbbmlFt_IOqvFlmLednp8b1ck/edit#gid=0 ===== Instructions ===== Step 1 - stop services and dismount volumes on prdapp01 (a) ssh expprodl@10.166.152.11 (b) stop god and apache on prdapp01 - 10.166.152.11 /etc/init.d/god stop (As user expprodl) ~/bin/apache_sss.sh stop (As user expprodl) (C) su root (d) umount the following mount points umount /deploy/crossbow/shared/assets umount /deploy/crossbow/shared/system/uploaded_datas umount /deploy/crossbow/shared/system/images umount /deploy/crossbow/shared/system/images umount /deploy/crossbow/shared/system/sheets umount /deploy/crossbow/shared/system/site_backgrounds umount /deploy/crossbow/shared/system/site_logos umount /deploy/crossbow/shared/system/styles umount /deploy/crossbow/shared/system/templates umount /deploy/crossbow/shared/streams Step (2) - stop services and dismount volumes on prdapp02 (a) ssh expprodl@10.166.152.19 (b) stop god and apache on prdapp02 - 10.166.152.19 /etc/init.d/god stop (As user expprodl) ~/bin/apache_sss.sh stop (As user expprodl) (c) su root (d) umount below volumes umount /deploy/crossbow/shared/assets umount /deploy/crossbow/shared/system/uploaded_datas umount /deploy/crossbow/shared/system/images umount /deploy/crossbow/shared/system/images umount /deploy/crossbow/shared/system/sheets umount /deploy/crossbow/shared/system/site_backgrounds umount /deploy/crossbow/shared/system/site_logos umount /deploy/crossbow/shared/system/styles umount /deploy/crossbow/shared/system/templates umount /deploy/crossbow/shared/streams Verify that the maintenance page is up. Step (3)- upgrade GlusterFS server from 3.2.4 to 3.5 in db slave box (a) ssh expprodl@10.166.152.13 (b) su root (c) /etc/init.d/glusterfs stop stop glusterd # nfs processes etc. sudo kill `pgrep gluster` sudo portmap stop (d) backup the following mount points cd /data00/ tar -pcvzf /backup_glusterfs/assets_20140728.tar.gz assets/ tar -pcvzf /backup_glusterfs/images_20140728.tar.gz images/ tar -pcvzf /backup_glusterfs/sheets_20140728.tar.gz sheets/ tar -pcvzf /backup_glusterfs/site_backgrounds_20140728.tar.gz site_backgrounds/ tar -pcvzf /backup_glusterfs/site_logos_20140728.tar.gz site_logos/ tar -pcvzf /backup_glusterfs/styles_20140728.tar.gz styles/ tar -pcvzf /backup_glusterfs/templates_20140728.tar.gz templates/ tar -pcvzf /backups_glusterupg/sharedpub_20140728.tar.gz sharedpub/ (e) cd /etc tar cvzf glusterd_26_Jul_2014.tar.gz glusterd tar cvzf glusterfs_26_Jul_2014.tar.gz glusterfs (this does not seem to be a valid configuration folder) (f) upgrade glusterfs on dbslave step f1 compile - install gluster - did not work mkdir /opt/src/glusterfs cd /opt/src/glusterfs wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/glusterfs-3.5.0.tar.gz tar zxvf glusterfs-3.5.0.tar.gz cd glusterfs-3.5.0 ./autogen.sh #might prompt to install dependencies make and make install Used the rpms made by gluster.org http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.0/CentOS/epel-5Server/x86_64/glusterfs-libs-3.5.0-2.el5.x86_64.rpm http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.0/CentOS/epel-5Server/x86_64/glusterfs-fuse-3.5.0-2.el5.x86_64.rpm http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.0/CentOS/epel-5Server/x86_64/glusterfs-server-3.5.0-2.el5.x86_64.rpm http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.0/CentOS/epel-5Server/x86_64/glusterfs-3.5.0-2.el5.x86_64.rpm step f2 setfattr -n trusted.glusterfs.volume-id -v 0x5e957f5d47c64509b0651e0c1123d286 /data00/assets setfattr -n trusted.glusterfs.volume-id -v 0xe1015abf7ce34697ab0b94b9a1558d7e /data00/sharedpub setfattr -n trusted.glusterfs.volume-id -v 0x233515755b144eee8d95621199d121bb /data00/images setfattr -n trusted.glusterfs.volume-id -v 0x7fba152b45b5458d8d002fab012257a7 /data00/sheets setfattr -n trusted.glusterfs.volume-id -v 0xd7c0e1d7581e4449a1ce00ec2a8f336c /data00/site_backgrounds setfattr -n trusted.glusterfs.volume-id -v 0x4b1017c9d7d5400282f5453e6fb2045c /data00/site_logos setfattr -n trusted.glusterfs.volume-id -v 0xd31fdc7fcc7f4f1c9964f392f5f94846 /data00/styles setfattr -n trusted.glusterfs.volume-id -v [volid] /data00/templates step f3 gluster volume status killall glusterd glusterd --xlator-option *.upgrade=on -N (did not this step for RPM upgrade as in gluster.org notes /Vijay Bellur) start glusterd Step 4 - upgrade glusterfs client on prdapp02 - 10.166.152.19 cd /etc/yum.repos.d/ wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/CentOS/glusterfs-epel.repo.el5 mv glusterfs-epel.repo.el5 glusterfs-epel.repo yum install glusterfs-client test glusterfs connectivity - app server02 to db slave mkdir /tmp/a mount.glusterfs 10.166.152.13:/site_logos /tmp/a umount /tmp/a Step 5 - upgrade glusterfs client on prdapp01 - 10.166.152.11 ssh expprodl@10.166.152.11 su root cd /etc/yum.repos.d/ wget http://download.gluster.org/pub/gluster/glusterfs/3.5/LATEST/CentOS/glusterfs-epel.repo.el5 mv glusterfs-epel.repo.el5 glusterfs-epel.repo yum install glusterfs-client test glusterfs connectivity - app server01 to db slave mkdir /tmp/a mount.glusterfs 10.166.152.13:/site_logos /tmp/a umount /tmp/a step 6 - upgrade glusterfs server on prdcollab01 (ubuntu) - 10.166.152.17 step 6.1 - stop BBB on prdcollab01 - 10.166.152.17 ssh expbbbp@10.166.152.17 sudo su - /etc/init.d/glusterfs stop step 6.2 - upgrade glusterfs server on prdcollab01 - 10.166.152.17 Stop all glusterd processes. stop glusterd # nfs processes etc. sudo kill `pgrep gluster` sudo portmap stop Take a backup of /etc/glusterd. backup -/usr/share/red5/webapps/video/streams Add PPA and Install Gluster 3.5 add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5 apt-get install gluster-server gluster-client The installation process will move /etc/glusterd/* to /var/lib/glusterd/*. Verify directory structure. New files might be added, thats OK. If the previous step fails (i.e. files were not migrated), do a manual copy. mkdir -p /var/lib/glusterd cp -r /etc/glusterd/* /var/lib/glusterd/ Start glusterd in upgrade mode. It will upgrade existing volumes' index files to support new version. killall glusterd glusterd --xlator-option *.upgrade=on -N start glusterd setfattr -n truster.glusterfs.volume-id -v 0x (info file has the file attr) symlink /usr/share/red5/streams to /opt/streams step 7 Reboot prddbslav01\\ step 8 Reboot prdcollab01\\ step 9 verify Gluster server services\\ step 10 check ps -ef | grep gluster on all the boxes and compare it against the output already saved\\ step 11 Reboot prdapp01\\ step 12 Reboot prdapp02\\ Step 13 - Mount all endpoints on app1 and app2 (if they are not already mounted after the reboot) mount.glusterfs 10-166-152-13:/asset-volume /deploy/crossbow/shared/assets mount.glusterfs 10-166-152-13:/data-volume /deploy/crossbow/shared/system/uploaded_datas mount.glusterfs 10-166-152-13:/images-volume /deploy/crossbow/shared/system/images mount.glusterfs 10-166-152-13:/site_backgrounds /deploy/crossbow/shared/system/site_backgrounds mount.glusterfs 10-166-152-13:/site_logos /deploy/crossbow/shared/system/site_logos mount.glusterfs 10-166-152-13:/styles /deploy/crossbow/shared/system/styles mount.glusterfs 10-166-152-13:/sheets /deploy/crossbow/shared/system/sheets mount.glusterfs 10-166-152-13:/templates /deploy/crossbow/shared/system/templates mount.glusterfs prodcollab01.learninguniver.se:/streams /deploy/crossbow/shared/streams Set the file permissions appropriately at the top level of the mount on prdapp01 / prdapp02 Step 14 test (a) Put app02 in LB maintanance mode ssh root@10.166.152.16 echo "enable server learnexa/prodapp02" | socat stdio /var/run/haproxy/haproxy.sock (b) Bring app02 service up and test. /etc/init.d/god stop (As user expprodl) ~/bin/apache_sss.sh stop (As user expprodl) IF app02 testing is fine , Test app01 like (a) and (b) IF both app01 and app02 are fine go to 15a ELSE go to 15b Step 15a - Bring the site up ssh root@10.166.152.16 echo "enable server learnexa/prodapp01" | socat stdio /var/run/haproxy/haproxy.sock echo "enable server learnexa/prodapp02" | socat stdio /var/run/haproxy/haproxy.sock Step 15b - Rollback Rollback - Glusterfs server/client on prddb-slave(centos) Uninstall glusterserver 3.5.0/1 Uninstall glusterclient 3.5.0/1 install glusterserver 3.2.4 (rpm for core and fuse are in dbslave - /opt/src/glusterfs) install glusterclient 3.2.4 restore backed up /etc/gluster restore backed up /data00/folders setfattr if needed (but do not think it is necessary) reboot verify glusterfs services are started Rollback - Glusterfs server on prdcollab01(ubuntu) Uninstall glusterserver 3.5.0/1 Uninstall glusterclient 3.5.0/1 install glusterserver 3.2.4(ubuntu version / source needs to be searched out) install glusterclient 3.2.4(ubuntu version / source needs to be searched out) restore backed up /etc/gluster restore backed up /streams mount setfattr if needed (but do not think it is necessary) reboot verify glusterfs services are started Rollback - Glusterfs client on prdapp01(centos) uninstall glusterclient 3.5.0/1 install glusterclient 3.2.4 (centos rpm available in prdb-slave /opt/glusterfs) reboot mount all the volumes (not necessary to do it manually - mtab should take care - but mount commands with mount points are documented above) Rollback - Glusterfs client on prdapp02(centos) uninstall glusterclient 3.5.0/1 install glusterclient 3.2.4 reboot mount all the volumes (not necessary to do it manually - mtab should take care - but mount commands with mount points are documented above)