** This page was rewritten on 04/01/2014. Please use the revisions feature to access old version. ** ==== Check, Install, Configure the below on Web/App Server ==== ** Follow the instructions on [[setup_an_app_server|Setup an app server]] ** ==== Download and Install MySQL database on database server ==== ** Follow the instructions on [[setup_db_server|Setup db server]] ** ==== Install and Configure Big Blue Button ==== * [[setup_bbb_server| Setup BBB Server]] * [[BigBlueButton Integration]] ==== Install Capistrano on build server (or server running cap tasks) ==== gem install capistrano -v 2.5.21 gem install capistrano-ext -v 1.2.1 == Check,update the deploy.rb script on build server (or server running cap tasks) == set :scm_username, 'svnuser' set :scm_password, 'svnpassword' set :deploy_via, "export" set :deploy_to, "/deploy/crossbow" set :user, "ssh_targetsrvr_user" set :local_repository, "reposurl" (-- repository as seen from build server or cap server, if omitted, same as :repository) set :repository, "reposurl" (-- repository as seen from target server if local_repository is defined, else common for build and target) Add tasks/run commands where necessary, for eg: below added under customs:symlink to create links to database.yml and production.rb after code update) run "ln -nfs /deploy/database.yml #{release_path}/config/database.yml" run "ln -nfs /deploy/production.rb #{release_path}/config/environments/production.rb" In the respective /deploy/crossbow/current/config/environments/{#env}.rb file, add the below line for conversion of pdf2swf: Paperclip.options[:command_path] = "/usr/local/bin/" ===== Deployment Steps ===== ==== Cold Deploy ==== Run cold deploy for the first time deployment of the application to the target server cap production deploy:setup cap production deploy:check cap production deploy:cold ==== Subsequent deployments ==== For applications that have already been deployed once, run cap production deploy cap production deploy:migrate Capistrano "before" and "after" hooks can be added in deploy.rb to fine tune the deployment process. ==== Knowing the deployment scripts ==== === runcap.sh === Values to review/edit in runcap.sh: CAPDIR (current capify dir, where runcap.sh and config/* reside) MAILRECIP (recipient list for cap output) Questions to be answered for runcap.sh Select Target environment (select any one number): This is where the code needs to be pushed Select Cap Task (select any one number): Capistrano task to be ran from the displayed list, after first cold deploy, you'll most likely select just the deploy option. * Versions prior to 1.5.5.0-FRED use TAG value, refer to older revision may 16 2014 of this document * Enter BRANCH value (will be prompted if Cap Task is deploy or deploy:cold): This is the branch value where the final tested code has been checked out and branched by QA/Release Team. If left empty, capistrano will point to the "trunk" for repository. https://repos.exphosted.com/svnrepos/crossbow/trunk will be used as repository, if BRANCH is empty https://repos.exphosted.com/svnrepos/crossbow/branches/#{BRANCH} will be used as repository, if BRANCH value is entered Ready to run task (y or Y): Final confirmation to run the cap script, enter y or Y Enter SVN UserName: (if Cap Task is deploy or deploy:cold, ideally would want to have a read-only user for repos) Enter SVN Password: (if Cap Task is deploy or deploy:cold) BEGIN CAUTION If the SVN credentials are incorrect, the script will seem to be doing nothing. It should show string "SVNOUT is ", with successful credentials, if you don't see this within few seconds, hit "Enter" couple times, as there won't be any valid credentials prompt. Let go the password prompts (shown below) to end the script. Password for '': Password for '': SVNOUT is Authentication realm: Subversion Repository Username: Authentication realm: Subversion Repository Username: svn: PROPFIND request failed on '/svnrepos/crossbow/trunk' svn: PROPFIND of '/svnrepos/crossbow/trunk': authorization failed (https://repos.exphosted.com) *** [err :: SVN credentials or SVNURL https://repos.exphosted.com/svnrepos/crossbow/trunk is invalid.. === Errors encountered during the load, error file will be emailed.. ==== END CAUTION Choosing any task will email the standard out and error (if any) logs to MAILRECIP, when the cap tasks are done. === config/deploy.rb === Values to review/edit in config/deploy.rb user (ssh user to app/db servers) coreurl (root url for repository, including project name) curlsrv (Application URL) svnrun.include? 'Rakefile' (output verification from svn list call, expecting Rakefile from svn list command on trunk or branch repo) curlvar1.include? 'Expertus Rocks!' (output verification from curl call in deploy:restart, after touching restart.txt, to re-invoke Passenger ApplicationSpawner) Cap Tasks (custom/edited) verify_tag (to verify BRANCH, sets the repository value to trunk or branch) solr (stop/start solr on App servers, before and after deploy) customs:dblink (to re-link database.yml on App servers, after deploy) deploy:restart (added curl call to Application URL, to re-invoke AppSpawner and look for 'Expertus Rocks' string; show output of Passenger ApplicationSpawner) web:disable (add iptable rules to block port 80 from lb) web:enable (stop iptable server to reenable port 80 for lb) === config/deply/staging.rb === Add/Edit/Review rails_env,db_env,app,web,db values === config/deply/production.rb === Add/Edit/Review rails_env,db_env,app,web,db values ==== Running the deployment scripts (Crossbow) ==== ** Notes for runcap.sh on PROD environment: ** == Before deploy == - Check and record the state of existing PROD deployments == Login to the build server as the normal user expprodl (not root). == Use option 1 to push incremental builds \\ Use option 8 to open app to the Internet (after the validation is completed from back end) After deployment - test: - whether you can login. - whether you are able to see catalog. - whether you are logged on to chat (green ball). Call the runcap.sh from the cap directory cd ~/crossbow (If the directory does not exist then you are not logged in as the correct user. DO NOT su - expprodl) ./runcap.sh Output for deploy task show below (your output may vary depending on options selected): ./runcap.sh Select Target environment 1). Staging 2). Production Enter target environment number [1,2]: 1 (Enter 1) Select Cap Task 1). deploy (Use this for incremental code deployments) 2). deploy:setup (Use this to setup the necessary application folders) 3). deploy:check (Use this to verify application folder setup) 4). deploy:cold 5). deploy:migrate 6). deploy:restart 7). deploy:update 8). deploy:web:enable (Use this to remove maintenance page, App becomes accessible only from back-end/VPN) 9). httpstate:allup Enter Cap Task number [1-9]: 1 (Enter 1) Enter BRANCH value or leave blank to use trunk as repository: Ready to run task deploy in environment staging..? [yY]: y (Enter y) Enter SVN UserName: Enter SVN Password: NORMAL OUTPUT from the cap script: * executing `deploy:web:disable' * executing "/deploy/systasks/web.sh disable" servers: ["192.168.1.215"] [192.168.1.215] executing command ** [out :: 192.168.1.215] Not required in QA ... command finished in 176ms triggering after callbacks for `deploy:web:disable' * executing `httpstate:down' * executing "/deploy/systasks/apache.sh down" servers: ["192.168.1.215"] [192.168.1.215] executing command ** [out :: 192.168.1.215] Not required in QA ... command finished in 93ms * executing `god:stop' * executing "/deploy/systasks/god.sh stop" servers: ["192.168.1.215"] [192.168.1.215] executing command ** [out :: 192.168.1.215] Sending 'stop' command ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] The following watches were affected: ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] workling ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] Sending 'stop' command ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] The following watches were affected: ** [out :: 192.168.1.215] memcached ** [out :: 192.168.1.215] Sending 'stop' command ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] The following watches were affected: ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] openoffice ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] Sending 'stop' command ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] The following watches were affected: ** [out :: 192.168.1.215] xvfb ** [out :: 192.168.1.215] Sending 'stop' command ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] The following watches were affected: ** [out :: 192.168.1.215] starling ** [out :: 192.168.1.215] Sending 'stop' command ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] ** [out :: 192.168.1.215] The following watches were affected: ** [out :: 192.168.1.215] rufus ** [out :: 192.168.1.215] Stopped god command finished in 11624ms * executing `solr:stop' * executing "/opt/tomcat6/bin/shutdown.sh" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 476ms executing locally: "svn info https://repos.exphosted.com/svnrepos/crossbow/trunk --username murali --password PASSWORD --no-auth-cache -rHEAD" command finished in 1799ms * executing "svn export -q --username murali --password PASSWORD --no-auth-cache -r1001 https://repos.exphosted.com/svnrepos/crossbow/trunk /deploy/crossbow/releases/20110824145053 && (echo 1001 > /deploy/crossbow/releases/20110824145053/REVISION)" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 91660ms * executing `deploy:finalize_update' * executing "chmod -R g+w /deploy/crossbow/releases/20110824145053" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 172ms * executing "rm -rf /deploy/crossbow/releases/20110824145053/log /deploy/crossbow/releases/20110824145053/public/system /deploy/crossbow/releases/20110824145053/tmp/pids &&\\\n mkdir -p /deploy/crossbow/releases/20110824145053/public &&\\\n mkdir -p /deploy/crossbow/releases/20110824145053/tmp &&\\\n ln -s /deploy/crossbow/shared/log /deploy/crossbow/releases/20110824145053/log &&\\\n ln -s /deploy/crossbow/shared/system /deploy/crossbow/releases/20110824145053/public/system &&\\\n ln -s /deploy/crossbow/shared/pids /deploy/crossbow/releases/20110824145053/tmp/pids" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 132ms * executing "find /deploy/crossbow/releases/20110824145053/public/images /deploy/crossbow/releases/20110824145053/public/stylesheets /deploy/crossbow/releases/20110824145053/public/javascripts -exec touch -t 201108241452.25 {} ';'; true" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 1451ms triggering after callbacks for `deploy:finalize_update' * executing `bundle:install' * executing "ls -x /deploy/crossbow/releases" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 114ms * executing "cd /deploy/crossbow/releases/20110824145053 && bundle install --gemfile /deploy/crossbow/releases/20110824145053/Gemfile --path /deploy/crossbow/shared/bundle --deployment --quiet --without development test" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 488ms triggering after callbacks for `deploy:update_code' * executing `customs:dblink' * executing "ln -nfs /deploy/crossbow/shared/config/database.yml /deploy/crossbow/releases/20110824145053/config/database.yml" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 107ms * executing "ln -nfs /deploy/crossbow/shared/config/staging.rb /deploy/crossbow/releases/20110824145053/config/environments/staging.rb" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 110ms * executing "ln -nfs /deploy/crossbow/shared/config/staging.god /deploy/crossbow/releases/20110824145053/config/staging.god" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 106ms * executing "ln -nfs /deploy/crossbow/shared/config/sunspot.yml /deploy/crossbow/releases/20110824145053/config/sunspot.yml" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 109ms * executing "ln -nfs /deploy/crossbow/shared/config/bigbluebutton.yml /deploy/crossbow/releases/20110824145053/config/bigbluebutton.yml" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 108ms * executing `deploy:migrate' triggering before callbacks for `deploy:migrate' * executing `masterdb:backup' * executing "~/bin/runbackup.sh" servers: ["192.168.1.215"] [192.168.1.215] executing command ** [out :: 192.168.1.215] Backing up cb_staging db... ** [out :: 192.168.1.215] cb_staging database dmp gzipped to file cb_staging_201108240752.sql.gz ... command finished in 380ms * executing "cd /deploy/crossbow/releases/20110824145053; bundle exec rake RAILS_ENV=staging db:migrate" servers: ["192.168.1.215"] [192.168.1.215] executing command ** [out :: 192.168.1.215] (in /deploy/crossbow/releases/20110824145053) command finished in 8997ms * executing `deploy:symlink' * executing "rm -f /deploy/crossbow/current && ln -s /deploy/crossbow/releases/20110824145053 /deploy/crossbow/current" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 115ms triggering after callbacks for `deploy:symlink' * executing `customs:symlink' * executing "ln -nfs /deploy/crossbow/shared/assets/photos /deploy/crossbow/releases/20110824145053/public/photos" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 104ms * executing "ln -nfs /deploy/crossbow/shared/assets/assets /deploy/crossbow/releases/20110824145053/public/assets" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 104ms * executing "ln -nfs /deploy/crossbow/shared/assets/homepage_features /deploy/crossbow/releases/20110824145053/public/homepage_features" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 107ms * executing `deploy:package_assets' triggering before callbacks for `deploy:package_assets' * executing `sass:update' * executing "cd /deploy/crossbow/current && rake sass:update" servers: ["192.168.1.215"] [192.168.1.215] executing command ** [out :: 192.168.1.215] (in /deploy/crossbow/releases/20110824145053) command finished in 9433ms * executing "cd /deploy/crossbow/current && rake asset:packager:build_all" servers: ["192.168.1.215"] [192.168.1.215] executing command ** [out :: 192.168.1.215] (in /deploy/crossbow/releases/20110824145053) ** [out :: 192.168.1.215] Created /deploy/crossbow/releases/20110824145053/public/javascripts/base_packaged.js ** [out :: 192.168.1.215] Created /deploy/crossbow/releases/20110824145053/public/javascripts/annotation_packaged.js ** [out :: 192.168.1.215] Created /deploy/crossbow/releases/20110824145053/public/stylesheets/base_packaged.css command finished in 6204ms * executing `god:start' * executing "/deploy/systasks/god.sh start" servers: ["192.168.1.215"] [192.168.1.215] executing command ** [out :: 192.168.1.215] starting god command finished in 5236ms * executing `solr:start' * executing "/opt/tomcat6/bin/startup.sh" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 162ms ** transaction: commit * executing `deploy:restart' * executing "touch /deploy/crossbow/current/tmp/restart.txt" servers: ["192.168.1.215"] [192.168.1.215] executing command command finished in 97ms Making curl call to http://qa01.exphosted.com ... Please wait ... ** [out :: Site http://qa01.exphosted.com is OK] **Now run the release specific tasks.** ==== Running the deployment scripts (BBB/Recorder) ==== **Before deploy** Make sure that release tasks have been executed for app server. **Login** Login to the respective BBB server (devcollab01/qacollab01/prodcollab01) as the provided BBB user (not root). **Deploy** For BBBClient (provide the BRANCH value, password for the repos user and finally the password for logged in user to restart bbb-conf with sudo) (Versions prior to 1.5.5.0-FRED use TAG value, refer to older revision may 16 2014 of this document *) cd ~/bin (If the directory does not exist, then most likely, you are not logged in as the correct user.) ./deploybbbclient.sh Enter BBB BRANCH value or leave blank to use trunk as repository: 0.9.2.0-LYNN Ready to deploy BBB code ..? [yY]: y Starting... Authentication realm: Subversion Repository Password for 'murali': ..... branding: [mxmlc] Loading configuration file /home/expbbbu/dev/tools/flex/frameworks/flex-config.xml [mxmlc] Required RSLs: [mxmlc] http://fpdownload.adobe.com/pub/swz/flex/4.5.0.20967/framework_4.5.0.20967.swz with 1 failover. [mxmlc] http://fpdownload.adobe.com/pub/swz/tlf/2.0.0.232/textLayout_2.0.0.232.swz with 1 failover. [mxmlc] http://fpdownload.adobe.com/pub/swz/flex/4.5.0.20967/mx_4.5.0.20967.swz with 1 failover. [mxmlc] /home/expbbbu/dev/srctrunk/bigbluebutton-client/bin/branding/css/live_event.css.swf (54452 bytes) BUILD SUCCESSFUL Total time: 9 seconds Done with deployment... Restarting all BBB processes... .... For recorder (provide the BRANCH value, password for the repos user) cd ~/bin ./deploy_recorder_server.sh Enter Recorder Server BRANCH value or leave blank to use trunk as repository: 1.5.5.0-FRED Ready to deploy Recorder Server code ..? [yY]: y Starting... Authentication realm: Subversion Repository Password for 'murali': .... compile: [echo] java.home is /usr/lib/jvm/java-6-openjdk/jre and the target version is 1.6 [echo] red5.root is /usr/share/red5 [echo] javac version: 1.6.0_20 [javac] Compiling 1 source file to /home/expbbbu/dev/source/recorder/www/WEB-INF/classes war: [war] Building war: /home/expbbbu/dev/source/recorder/dist/recorder.war BUILD SUCCESSFUL Total time: 2 seconds Buildfile: build.xml Main targets: retrieve Retrieves the libraries if needed Default target: war * Stopping Red5 Server red5 [ OK ] * Stopping Tomcat servlet engine tomcat6 .... **After deploy** - Test whether you can start a Live Event - Test whether you can participate and invite others in the Live Event. ===== Performance Tuning ===== ==== Phusion Passenger ==== PassengerStatThrottleRate 10 (default 0) ### checks restart.txt and few other config .rb files ### filesystem checks will be performed at most once every 10 seconds with above value ### add in global section of httpd.conf PassengerPreStart http://learninguniver.se/ ### pre-start application instances during Apache startup ### add in global section of httpd.conf Note: File /usr/local/lib/ruby/gems/1.8/gems/passenger-3.0.7/helper-scripts/prespawn changed as under for the above to work with VirtualHost defined with IP address instead of *:port (Not required for passenger-3.0.8) socket = TCPSocket.new('127.0.0.1', uri.port) == old value socket = TCPSocket.new('learninguniver.se', uri.port) == new value EDIT: /opt/apache2/bin/envvars to include ruby path PATH=/opt/ruby-enterprise-1.8.7-2011.03/bin:$PATH export PATH PassengerPoolIdleTime 0 (default 300 sec) ### When this value is set to 0, application instances will not be shutdown unless it’s really necessary ### add in global section of httpd.conf RailsFrameworkSpawnerIdleTime 0 (default 1800 sec) ### FrameworkSpawner server will never idle timeout ### may not be effective, since default PassengerSpawnMethod = smart-lv2 which skips the framework spawner and uses the application spawner directly. ### add in global section of httpd.conf RailsAppSpawnerIdleTime 0 (default 600 sec) ### ApplicationSpawner server will never idle timeout. ### add in global section of httpd.conf RailsAutoDetect off ### add in global section of httpd.conf RailsBaseURI / ### add in Virtual Host section ==== BBB Components ==== === Red5 === Edit /etc/init.d/red5 to include/change as per below: GCLOG=/usr/share/red5/log/red_gc_`date '+%Y_%m_%d_%H%M'`.log export GCLOG JAVA_OPTS="-Dpython.home=lib -Xms768m -Xmx768m -XX:NewRatio=3 -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SurvivorRatio=4 -XX:+DisableExplicitGC -Xnoclassgc -Xloggc:${GCLOG} -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dred5.root=$RED5_HOME -Dlogback.ContextSelector=org.red5.logging.LoggingContextSelector -Dcatalina.useNaming=true -Djava.security.debug=failure -cp $RED5_CLASSPATH org.red5.server.Bootstrap" This will log GC details in file /usr/share/red5/log/red_gc_YYYY_MM_DD_HHMI.log. Adjust Xms, Xmx and MaxPermSize as demand/memory permit. === ActiveMQ === Edit /etc/init.d/activemq to include/change as per below: GCLOG=/usr/share/activemq/logs/activemq_gc_`date '+%Y_%m_%d_%H%M'`.log export GCLOG JAVA_OPTS="$JAVA_OPTS -Xms768m -Xmx768m -XX:NewRatio=3 -XX:PermSize=128m -XX:MaxPermSize=128m -XX:SurvivorRatio=4 -XX:+DisableExplicitGC -Xnoclassgc -Xloggc:${GCLOG} -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Dactivemq.classpath=$ACTIVEMQ_HOME/conf -Dactivemq.home=$ACTIVEMQ_HOME -Dactivemq.base=$ACTIVEMQ_HOME -jar $ACTIVEMQ_HOME/bin/run.jar" This will log GC details in file /usr/share/activemq/logs/activemq_gc_YYYY_MM_DD_HH_MI.log. Adjust Xms, Xmx and MaxPermSize as demand/memory permit. ===== Adhoc Notes ===== ==== Chat Server ==== Listens on Port: 8090 (instead of 8080) Current version of juggernaut seemed to have issue starting with node.\\ To use the older version, use the saved {{:juggernaut.tgz|juggernaut.tgz}} file ==== Installing Solr/Tomcat On Search Server ==== wget http://apache.mirrors.hoobly.com/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz tar -xzf apache-tomcat-6.0.32.tar.gz mv apache-tomcat-6.0.32 /opt/tomcat6 wget http://mirror.atlanticmetro.net/apache//lucene/solr/1.4.1/apache-solr-1.4.1.tgz tar -xzf apache-solr-1.4.1.tgz cd apache-solr-1.4.1 cp -r example/solr /opt/ cp dist/apache-solr-1.4.1.war /opt/solr/ cp dist/apache-solr-1.4.1.war /opt/tomcat6/webapps/solr.war Set solr data path in /opt/solr/conf/solrconfig.xml (set to the shared drive on search server, unsure if this is required) ${solr.data.dir:/nfs00/share00/solr/data} mkdir -p /nfs00/share00/solr/data /opt/tomcat6/bin/catalina.sh (set JAVA_OPT under the while loop) JAVA_OPTS="$JAVA_OPTS -Dsolr.solr.home=/opt/solr" export JAVA_OPTS /opt/tomcat6/conf/tomcat-users.xml to include user/passwd UTF-8 Support: /opt/tomcat6/conf/server.xml, add URIEncoding="UTF-8" Schema.xml Copy schema.xml from /deploy/crossbow/current/solr/conf (src = App server) to /opt/solr/conf (destination = Tomcat/Search server) Change ownership of /opt/solr and /opt/tomcat6 to the app user used to run tomcat.