Hotfix release available: 2025-05-14b "Librarian". upgrade now! [56.2] (what's this?)
Hotfix release available: 2025-05-14a "Librarian". upgrade now! [56.1] (what's this?)
New release available: 2025-05-14 "Librarian". upgrade now! [56] (what's this?)
Hotfix release available: 2024-02-06b "Kaos". upgrade now! [55.2] (what's this?)
Hotfix release available: 2024-02-06a "Kaos". upgrade now! [55.1] (what's this?)
New release available: 2024-02-06 "Kaos". upgrade now! [55] (what's this?)
Hotfix release available: 2023-04-04b "Jack Jackrum". upgrade now! [54.2] (what's this?)
Hotfix release available: 2023-04-04a "Jack Jackrum". upgrade now! [54.1] (what's this?)
New release available: 2023-04-04 "Jack Jackrum". upgrade now! [54] (what's this?)
Hotfix release available: 2022-07-31b "Igor". upgrade now! [53.1] (what's this?)
Hotfix release available: 2022-07-31a "Igor". upgrade now! [53] (what's this?)
New release available: 2022-07-31 "Igor". upgrade now! [52.2] (what's this?)
New release candidate 2 available: rc2022-06-26 "Igor". upgrade now! [52.1] (what's this?)
New release candidate available: 2022-06-26 "Igor". upgrade now! [52] (what's this?)
Hotfix release available: 2020-07-29a "Hogfather". upgrade now! [51.4] (what's this?)
New release available: 2020-07-29 "Hogfather". upgrade now! [51.3] (what's this?)
New release candidate 3 available: 2020-06-09 "Hogfather". upgrade now! [51.2] (what's this?)
New release candidate 2 available: 2020-06-01 "Hogfather". upgrade now! [51.1] (what's this?)
New release candidate available: 2020-06-01 "Hogfather". upgrade now! [51] (what's this?)
Hotfix release available: 2018-04-22c "Greebo". upgrade now! [50.3] (what's this?)
Hotfix release available: 2018-04-22b "Greebo". upgrade now! [50.2] (what's this?)
production_reboot_checklist

Production server Reboot check list

Component List

Server Role Component Restarted during reboot Other comments
Font end server Haproxy YES
Front end/Chat redisYesChat component
Front end/Chat nodeYesChat component
Front end Haproxy ip blocking script YES Needed to block attackers
App servers ApacheYes This will start passenger instances.
App servers PassengerYesPassenger ApplicationSpawner gets restarted by touching tmp/restart.txt
App servers godYesBelow processes are also restarted by god. Should be started as the app user (expprodl)
App servers memcachedYesRestarted by god
App servers XvfbYesRestarted by god
App servers openofficeYesRestarted by god
App servers rufusYesRestarted by god
App servers mounts & permissions Yes Mount points are fine as of now but not permissions(7090 - Fixed).
DB slave solrYesTomcat process for Solr is restarted
DB Slave MYSQLYesSlave DB
DB slave Gluster server Yes Needed for assets mounting
DB Master MySQLYesMaster DB
Collab Server BBB Yes
Collab Server Gluster server Yes Needed for streams mounting

Apache

Look for at least few apache processes running:

/opt/apache2/bin/httpd -k start

Passenger

Passenger processes are started in the following sequence (with start/restart of Apache web server)

Apache–>PassengerWatchdog–>PassengerHelperAgent–>Passenger spawn server–>“Passenger ApplicationSpawner: /deploy/crossbow/current”

PassengerLoggingAgent is also started by PassengerWatchdog

Ensure the below passenger processes are running:

				 
PassengerWatchdog
PassengerHelperAgent
Passenger spawn server                                                                                        
PassengerLoggingAgent

Below process is the only one restarted by the Application Deployment (on touch tmp/restart.txt)

Passenger ApplicationSpawner: /deploy/crossbow/current

God

Look for the below process, ensure its recently started:

/deploy/crossbow/god.sh status #To check the status (run as expprodl user)
/deploy/crossbow/god.sh start #To start (run as expprodl user)

Processes controlled by god

Look for below processes (started by god), ensure they are all recently started:

Memcached:

memcached -d -m 3000

Xvfb:

Xvfb :1 -screen 0 800x600x24

Openoffice:

/usr/bin/soffice.bin -headless -accept=socket,host=127.0.0.1,port=8100;urp; -nofirststartwizard

Rufus:

ruby /deploy/crossbow/current/script/rufus

Chat

Chat processes are NOT restarted during reboot, ensure they are running:

~/bin/chat_sss.sh status (To check the status, as expprodl)
~/bin/chat_sss.sh start (To start, as expprodl)

Solr

Look for the below Tomcat_Solr (java process), ensure its recently started:

 ~/bin/tomcat_sss.sh status # Check status, as user expprodl
~/bin/tomcat_sss.sh start # start, as user expprodl

MySQL

To be updated

Check the follwing on both app servers (app01 and app02)

1) Mount points (mounted from a shared drive):

 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       57G   25G   29G  47% /
/dev/sda1              99M   40M   55M  43% /boot
tmpfs                 7.9G     0  7.9G   0% /dev/shm
glusterfs#prodcollab01.learninguniver.se:/streams
                       60G   22G   35G  39% /deploy/crossbow/shared/streams
glusterfs#10-166-152-13:/asset-volume
                      493G  172G  295G  37% /deploy/crossbow/shared/assets
glusterfs#10-166-152-13:/images-volume
                      493G  172G  295G  37% /deploy/crossbow/shared/system/images
glusterfs#10-166-152-13:/site_backgrounds
                      493G  172G  295G  37% /deploy/crossbow/shared/system/site_backgrounds
glusterfs#10-166-152-13:/site_logos
                      493G  172G  295G  37% /deploy/crossbow/shared/system/site_logos
glusterfs#10-166-152-13:/styles
                      493G  172G  295G  37% /deploy/crossbow/shared/system/styles
glusterfs#10-166-152-13:/sheets
                      493G  172G  295G  37% /deploy/crossbow/shared/system/sheets
glusterfs#10-166-152-13:/templates
                      493G  172G  295G  37% /deploy/crossbow/shared/system/templates

2) Links under /deploy/crossbow/current/public/system folder:

ls -la
total 68
drwxr-xr-x  12 expprodl expprodl  4096 Feb  5 20:37 .
drwxr-xr-x  13 expprodl expprodl  4096 Jan 23 21:13 ..
drwxr-xr-x   3 expprodl expprodl  4096 Feb  5 20:37 idp_certificates
drwxr-xr-x 729 expprodl expprodl 20480 Apr  4 20:07 images
-rw-r--r--   1 expprodl expprodl  1260 Feb  6  2014 maintenance.html
drwxr-xr-x  88 expprodl expprodl  4096 Apr  3 11:49 sheets
drwxr-xr-x   5 expprodl expprodl  4096 Jul 28  2014 site_backgrounds
drwxr-xr-x   5 expprodl expprodl  4096 Jul 28  2014 site_logos
drwxr-xr-x   3 expprodl expprodl  4096 Mar 26 23:44 styles
drwxr-xr-x  86 expprodl expprodl  4096 Apr  4 20:03 templates
drwxr-xr-x  10 expprodl expprodl  4096 Apr  5 12:42 uploaded_datas

After each reboot, verify that the links are valid, and contents under shared folder are the same (this ensures the shares are mounted and not stale mounts)

3) Besides, we also mount the /deploy/crossbow/shared/streams folder from the BBB server on each app node. After deploying BBB client and recorder code, verify on each node that this folder is in good mount state and not stale.

Commands reference

Validate Site Functionality

Create/login the test site - test.learnexa.com - and validate that all functions are working

Site Name: test (test.learnexa.com)
Admin user: test@learnexa.com
          password: Xr0ssb0w   (X is uppercase and 0 = zero)

Testing behind VPN

The maintenance page is set for external (Internet) users during code deployment. This can also be set active by running runcap script with option 9) deploy:web:disable. During this, site can still be accessed inside VPN.

Login through the VPN using the username/password provided.

Edit your (local) hosts file to include the below IP addresses pointing to the respective sites.

10.166.152.11   learnexa.com 
10.166.152.11   test.learnexa.com
#use 10.166.152.19 to test app server 2.

Begin testing (using domain names as usual), once done testing, comment out the above entries from your hosts file.

Maintenance page will be removed if things look correct and the site is ready to be released to the customers by selecting option 8) deploy:web:enable as Cap Task while running runcap.sh script. THIS FEATURE (of maintenance page) IS ONLY AVAILABLE ON PROD SETUP.

production_reboot_checklist.txt · Last modified: 2018/08/31 16:16 (external edit)