===== Production server Reboot check list =====
==== Component List ====
^ Server Role ^ Component ^ Restarted during reboot ^ Other comments ^
|Font end server | Haproxy | YES| |
|Front end/Chat |redis|Yes|Chat component|
|Front end/Chat |node|Yes|Chat component|
|Front end |Haproxy ip blocking script| YES| Needed to block attackers|
|App servers |Apache|Yes| This will start passenger instances. |
|App servers |Passenger|Yes|Passenger ApplicationSpawner gets restarted by touching tmp/restart.txt|
|App servers |god|Yes|Below processes are also restarted by god. Should be started as the app user (expprodl)|
|App servers |memcached|Yes|Restarted by god|
|App servers |Xvfb|Yes|Restarted by god|
|App servers |openoffice|Yes|Restarted by god|
|App servers |rufus|Yes|Restarted by god|
|App servers |mounts & permissions| Yes | Mount points are fine as of now but not permissions([[https://tracker.exphosted.com/view.php?id=7090|7090 - Fixed]]).|
|DB slave |solr|Yes|Tomcat process for Solr is restarted|
|DB Slave |MYSQL|Yes|Slave DB|
|DB slave | Gluster server| Yes | Needed for assets mounting|
|DB Master |MySQL|Yes|Master DB|
|Collab Server| BBB| Yes| |
|Collab Server| Gluster server| Yes | Needed for streams mounting |
==== Apache ====
Look for at least few apache processes running:
/opt/apache2/bin/httpd -k start
==== Passenger ====
Passenger processes are started in the following sequence (with start/restart of Apache web server)
Apache-->PassengerWatchdog-->PassengerHelperAgent-->Passenger spawn server-->"Passenger ApplicationSpawner: /deploy/crossbow/current"
PassengerLoggingAgent is also started by PassengerWatchdog
Ensure the below passenger processes are running:
PassengerWatchdog
PassengerHelperAgent
Passenger spawn server
PassengerLoggingAgent
Below process is the only one restarted by the Application Deployment (on touch tmp/restart.txt)
Passenger ApplicationSpawner: /deploy/crossbow/current
==== God ====
Look for the below process, ensure its recently started:
/deploy/crossbow/god.sh status #To check the status (run as expprodl user)
/deploy/crossbow/god.sh start #To start (run as expprodl user)
==== Processes controlled by god ====
Look for below processes (started by god), ensure they are all recently started:\\
Memcached:
memcached -d -m 3000
Xvfb:
Xvfb :1 -screen 0 800x600x24
Openoffice:
/usr/bin/soffice.bin -headless -accept=socket,host=127.0.0.1,port=8100;urp; -nofirststartwizard
Rufus:
ruby /deploy/crossbow/current/script/rufus
==== Chat ====
Chat processes are NOT restarted during reboot, ensure they are running:\\
~/bin/chat_sss.sh status (To check the status, as expprodl)
~/bin/chat_sss.sh start (To start, as expprodl)
==== Solr ====
Look for the below Tomcat_Solr (java process), ensure its recently started:
~/bin/tomcat_sss.sh status # Check status, as user expprodl
~/bin/tomcat_sss.sh start # start, as user expprodl
==== MySQL ====
To be updated
==== Shared folders and link ====
Check the follwing on both app servers (app01 and app02)
1) Mount points (mounted from a shared drive):
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
57G 25G 29G 47% /
/dev/sda1 99M 40M 55M 43% /boot
tmpfs 7.9G 0 7.9G 0% /dev/shm
glusterfs#prodcollab01.learninguniver.se:/streams
60G 22G 35G 39% /deploy/crossbow/shared/streams
glusterfs#10-166-152-13:/asset-volume
493G 172G 295G 37% /deploy/crossbow/shared/assets
glusterfs#10-166-152-13:/images-volume
493G 172G 295G 37% /deploy/crossbow/shared/system/images
glusterfs#10-166-152-13:/site_backgrounds
493G 172G 295G 37% /deploy/crossbow/shared/system/site_backgrounds
glusterfs#10-166-152-13:/site_logos
493G 172G 295G 37% /deploy/crossbow/shared/system/site_logos
glusterfs#10-166-152-13:/styles
493G 172G 295G 37% /deploy/crossbow/shared/system/styles
glusterfs#10-166-152-13:/sheets
493G 172G 295G 37% /deploy/crossbow/shared/system/sheets
glusterfs#10-166-152-13:/templates
493G 172G 295G 37% /deploy/crossbow/shared/system/templates
2) Links under /deploy/crossbow/current/public/system folder:
ls -la
total 68
drwxr-xr-x 12 expprodl expprodl 4096 Feb 5 20:37 .
drwxr-xr-x 13 expprodl expprodl 4096 Jan 23 21:13 ..
drwxr-xr-x 3 expprodl expprodl 4096 Feb 5 20:37 idp_certificates
drwxr-xr-x 729 expprodl expprodl 20480 Apr 4 20:07 images
-rw-r--r-- 1 expprodl expprodl 1260 Feb 6 2014 maintenance.html
drwxr-xr-x 88 expprodl expprodl 4096 Apr 3 11:49 sheets
drwxr-xr-x 5 expprodl expprodl 4096 Jul 28 2014 site_backgrounds
drwxr-xr-x 5 expprodl expprodl 4096 Jul 28 2014 site_logos
drwxr-xr-x 3 expprodl expprodl 4096 Mar 26 23:44 styles
drwxr-xr-x 86 expprodl expprodl 4096 Apr 4 20:03 templates
drwxr-xr-x 10 expprodl expprodl 4096 Apr 5 12:42 uploaded_datas
After each reboot, verify that the links are valid, and contents under shared folder are the same (this ensures the shares are mounted and not stale mounts)
3) Besides, we also mount the /deploy/crossbow/shared/streams folder from the BBB server on each app node.
After deploying BBB client and recorder code, verify on each node that this folder is in good mount state and not stale.
==== Commands reference ====
https://wiki.exphosted.com/doku.php/productionservers
https://wiki.exphosted.com/doku.php/qastageandproductiondeployment
===== Validate Site Functionality =====
Create/login the test site - test.learnexa.com - and validate that all functions are working
Site Name: test (test.learnexa.com)
Admin user: test@learnexa.com
password: Xr0ssb0w (X is uppercase and 0 = zero)
===== Testing behind VPN =====
The maintenance page is set for external (Internet) users during code deployment. This can also be set active by running runcap script with option 9) deploy:web:disable. During this, site can still be accessed inside VPN.
Login through the VPN using the username/password provided.
Edit your (local) hosts file to include the below IP addresses pointing to the respective sites.
10.166.152.11 learnexa.com
10.166.152.11 test.learnexa.com
#use 10.166.152.19 to test app server 2.
Begin testing (using domain names as usual), once done testing, comment out the above entries from your hosts file.
Maintenance page will be removed if things look correct and the site is ready to be released to the customers by selecting option 8) deploy:web:enable as Cap Task while running runcap.sh script. THIS FEATURE (of maintenance page) IS ONLY AVAILABLE ON PROD SETUP.