===== Production post deployment check list ===== ==== Component List ==== ^ Component ^ Restarted during upgrade ^ Other comments ^ |apache|Possible|With multiple nodes, look under the load blanacer maintenance page| |Passenger|Yes|Passenger ApplicationSpawner gets restarted by touching tmp/restart.txt| |god|Yes|Below processes are also restarted by god| |memcached|Yes|Restarted by god| |Xvfb|Yes|Restarted by god| |starling|Yes|Restarted by god| |workling|Yes|Restarted by god| |openoffice|Yes|Restarted by god| |rufus|Yes|Restarted by god| |redis|No|Chat component| |node|No|Chat component| |solr|Yes|Tomcat process for Solr is restarted, no other checks at this time| |MySQL|No|Only DB back is taken| ==== Apache ==== Look for at least few apache processes running: /opt/apache2/bin/httpd -k start ==== Passenger ==== Passenger processes are started in the following sequence (with start/restart of Apache web server) Apache-->PassengerWatchdog-->PassengerHelperAgent-->Passenger spawn server-->"Passenger ApplicationSpawner: /deploy/crossbow/current" PassengerLoggingAgent is also started by PassengerWatchdog Ensure the below passenger processes are running: PassengerWatchdog PassengerHelperAgent Passenger spawn server PassengerLoggingAgent Below process is the only one restarted by the Application Deployment (on touch tmp/restart.txt) Passenger ApplicationSpawner: /deploy/crossbow/current ==== God ==== Look for the below process, ensure its recently started: /opt/ruby-enterprise-1.8.7-2011.03/bin/ruby /opt/ruby-enterprise-1.8.7-2011.03/bin/god -c /deploy/crossbow/current/config/production.god -P /var/run/god/god.pid ==== Processes controlled by god ==== Look for below processes (started by god), ensure they are all recently started:\\ Memcached: memcached -d -m 1000 Xvfb: Xvfb :1 -screen 0 800x600x24 Starling: ruby /deploy/crossbow/shared/bundle/ruby/1.8/bin/starling -d -p 15151 -P /deploy/crossbow/current/tmp/pids/starling.pid -q /deploy/crossbow/current/log/ Workling: workling Openoffice: /usr/bin/soffice.bin -headless -accept=socket,host=127.0.0.1,port=8100;urp; -nofirststartwizard Rufus: ruby /deploy/crossbow/current/script/rufus ==== Chat ==== Chat processes are NOT restarted during the code upgrade, however, ensure they are running:\\ Redis: ./redis-server redis.conf Node: node server.js ==== Search ==== Look for the below Tomcat_Solr (java process), ensure its recently started: /usr/bin/java -Djava.util.logging.config.file=/opt/tomcat6/conf/logging.properties -Dsolr.solr.home=/opt/solr -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/tomcat6/endorsed -classpath /opt/tomcat6/bin/bootstrap.jar -Dcatalina.base=/opt/tomcat6 -Dcatalina.home=/opt/tomcat6 -Djava.io.tmpdir=/opt/tomcat6/temp org.apache.catalina.startup.Bootstrap start ==== MySQL ==== mysql server process is NOT restarted during the code upgrade.\\ Only the DB backup is taken before "rake db:migrate".\\ Look for backups under /data00/backups folder on master DB server (filename: DBNAME_YYYYMMDDHHMI.sql.gz) ==== Crossbow code ==== Ensure new folder is created under /deploy/crossbow/releases and the /deploy/crossbow/current folder is linked to it.\\ By default, the following files under current/config folder are linked to the respective names in /deploy/crossbow/shared/config/ bigbluebutton.yml\\ database.yml\\ recorder.yml\\ .god\\ sunspot.yml\\ environments/.rb While running the runcap.sh script to upgrade the code, if any errors are encountered by capistrano, a message as under is shown at the end of the script: === Errors encountered during the load, error file will be emailed.. ==== Error file production_cap_run_error.txt will be emailed, you can see the contents of this file in the same folder as runcap.sh script. ==== Shared folders and link ==== In order to keep persistent data between app nodes, we share some common folders between the nodes from a shared drive. The following folders are shared among the nodes (mounted from a shared drive): /deploy/crossbow/shared/assets /deploy/crossbow/shared/system/uploaded_datas /deploy/crossbow/shared/system/images Links under /deploy/crossbow/current/public folder: lrwxrwxrwx 1 expprodl expprodl 37 Feb 14 10:26 assets -> /deploy/crossbow/shared/assets/assets lrwxrwxrwx 1 expprodl expprodl 48 Feb 14 10:26 homepage_features -> /deploy/crossbow/shared/assets/homepage_features lrwxrwxrwx 1 expprodl expprodl 37 Feb 14 10:26 photos -> /deploy/crossbow/shared/assets/photos lrwxrwxrwx 1 expprodl expprodl 30 Feb 14 10:26 system -> /deploy/crossbow/shared/system After each deployment, verify that the links are valid, and contents under shared folder are the same (this ensures the shares are mounted and not stale mounts) Besides, we also mount the /deploy/crossbow/shared/streams folder from the BBB server on each app node. After deploying BBB client and recorder code, verify on each node that this folder is in good mount state and not stale. ==== Validate Site Functionality ==== Create/login the test site - test.learnexa.com - and validate that all functions are working Site Name: test (test.learnexa.com) Admin user: test@learnexa.com password: Xr0ssb0w (X is uppercase and 0 = zero) ===== Production post deployment testing ===== The maintenance page is set for external (Internet) users during code deployment. After code is deployed, maintenance page will still remain active till we test the site/features/functionality using the Internal IP address of the web sites. Login through the VPN using the username/password provided. Edit your (local) hosts file to include the below IP addresses pointing to the respective sites. 10.166.152.249 learnexa.com 10.166.152.250 test.learnexa.com 10.166.152.247 www.methodyoga.com Edit your (local) hosts file to include the below IP addresses pointing to the respective sites. 10.166.152.17 learnexa.com 10.166.152.14 test.learnexa.com 10.166.152.19 www.methodyoga.com Begin testing (using domain names as usual), once done testing, comment out the above entries from your hosts file. Maintenance page will be removed if things look correct and the site is ready to be released to the customers by selecting option 8) deploy:web:enable as Cap Task while running runcap.sh script. THIS FEATURE (of maintenance page) IS ONLY AVAILABLE ON PROD SETUP.