Time steps for deployment step 1 00:00:10 Vpn connect and login to prodapp01 step 2 00:00:50 maintenance page on step 3 00:00:30 mysql sync break step 4 Runcap.sh build deployment .(back fill tasks) **if build fails step 5 to step 8 applicable** step 5 00:02:00 database.yml fie change both app01 and app02 servers replace mysql master ip to slave ip step 6 00:00:05 stop mysql service slave server step 7 00:02:00 my.cnf change slave db as master server and start mysql server step 8 00:05:00 both app server restart passenger service step 9 00:00:40 remove maintenance page and site able to access 11.25 Total Time steps for rollback with mysqldump (Fail Back) option1 step 1 00:00:10 Vpn connect step 2 00:00:10 Drop database in mysql master step 3 00:03:00 create new db and import mysql dump step 4 Once mysql db identical master and slave (Based on db size time differ) step 5 00:03:00 swap my.cnf file in mysql master and slave servers and restart mysql service both servers step 6 00:00:10 login to prodapp01 step 7 00:00:50 maintenance page on step 8 00:05:00 database.yml fie change both app01 and app02 servers db masterip change step 9 00:00:40 both app server restart passenger service step 10 00:00:50 remove maintenance page 13.83 Total Time #!/bin/bash #title : mysql failover and failback #author : Sudharsan #Designation : Devops #10.166.152.15 MYSQL MASTER #10.166.152.23 MYSQL SLAVE #path /home/expprodl/bin/convert_as_master.sh mysql -u repl -p******** -h10.166.152.15 -P 3306 -e "STOP SLAVE;" mysql -u repl -p******** -h10.166.152.15 -P 3306 -e "RESET MASTER;" mysql -u repl -p******** -h10.166.152.23 -P 3306 -e "STOP SLAVE;" mysql -u repl -p******** -h10.166.152.23 -P 3306 -e "RESET MASTER;" ssh root@10.166.152.23 "/etc/init.d/mysqld stop" ssh root@10.166.152.15 "/etc/init.d/mysqld stop" echo "my.cnf changes done 10.166.152.15 MYSQL MASTER " ssh root@10.166.152.15 "cp /etc/.my.cnf-master /etc/my.cnf" #10.166.152.15 converted to master echo "my.cnf changes done 10.166.152.23 MYSQL SLAVE " ssh root@10.166.152.23 "cp /etc/.my.cnf-slave /etc/my.cnf" #10.166.152.23 converted to Slave ssh root@10.166.152.23 "/etc/init.d/mysqld start" ssh root@10.166.152.15 "/etc/init.d/mysqld start" echo "restart passenger on 10.166.152.14" ssh expprodl@10.166.152.14 "touch /deploy/crossbow/current/tmp/restart.txt" echo "restart passenger on 10.166.152.22" ssh expprodl@10.166.152.22 "touch /deploy/crossbow/current/tmp/restart.txt" sleep 30 echo "Please wait for a while passenger service restarting" #!/bin/bash #title : replication #description : This script automates the process of starting a Mysql Replication #author : Sudharsan #Designation : Devops #Date : 31-Jan-2017 #path /home/expprodl/bin/replication.sh DB=cbprod DUMP_FILE="/tmp/$DB-export-$(date +"%Y%m%d%H%M%S").sql" USER=repl PASS=******** MASTER_HOST=10.166.152.15 SLAVE_HOSTS=(10.166.152.23) echo "MASTER: $MASTER_HOST" mysql -h $MASTER_HOST "-u$USER" "-p$PASS" $DB <<-EOSQL & GRANT REPLICATION SLAVE ON *.* TO '$USER'@'%' IDENTIFIED BY '$PASS'; FLUSH PRIVILEGES; FLUSH TABLES WITH READ LOCK; DO SLEEP(3600); EOSQL echo " - Waiting for database to be locked" sleep 3 echo " - Dumping database to $DUMP_FILE" mysqldump -h $MASTER_HOST "-u$USER" "-p$PASS" --opt $DB > $DUMP_FILE echo " - Dump complete." MASTER_STATUS=$(mysql -h $MASTER_HOST "-u$USER" "-p$PASS" -ANe "SHOW MASTER STATUS;" | awk '{print $1 " " $2}') LOG_FILE=$(echo $MASTER_STATUS | cut -f1 -d ' ') LOG_POS=$(echo $MASTER_STATUS | cut -f2 -d ' ') echo " - Current log file is $LOG_FILE and log position is $LOG_POS" kill $! 2>/dev/null wait $! 2>/dev/null echo " - Master database unlocked" for SLAVE_HOST in "${SLAVE_HOSTS[@]}" do echo "SLAVE: $SLAVE_HOST" echo " - Creating database copy" mysql -h $SLAVE_HOST "-u$USER" "-p$PASS" -e "DROP DATABASE IF EXISTS $DB; CREATE DATABASE $DB;" scp $DUMP_FILE $SLAVE_HOST:$DUMP_FILE >/dev/null mysql -h $SLAVE_HOST "-u$USER" "-p$PASS" $DB < $DUMP_FILE echo " - Setting up slave replication" mysql -h $SLAVE_HOST "-u$USER" "-p$PASS" $DB <<-EOSQL & STOP SLAVE; CHANGE MASTER TO MASTER_HOST='$MASTER_HOST', MASTER_USER='$USER', MASTER_PASSWORD='$USER', MASTER_LOG_FILE='$LOG_FILE', MASTER_LOG_POS=$LOG_POS; START SLAVE; EOSQL # Wait for slave to get started and have the correct status sleep 2 # Check if replication status is OK SLAVE_OK=$(mysql -h $SLAVE_HOST "-u$USER" "-p$PASS" -e "SHOW SLAVE STATUS\G;" | grep 'Waiting for master') if [ -z "$SLAVE_OK" ]; then echo " - Error ! Wrong slave IO state." else echo " - Slave IO state OK" fi done ====== If Build fails on Shell script failover and fail back ====== If Build fails on shell script failover and fail back Steps Time connect open vpn 0:00:10 login prodapp01 0:00:10 runcap.sh 0:00:10 maintenance page on 0:00:50 mysql sync break 0:00:50 deploy build on app server 0:05:00 deploy build on collab server 0:11:00 rake task not included (if build fails) ====================SHELL SCRIPT WILL TAKE CARE START==================== stop slave mysql service stop master and slave changes in my.cnf (mysql) mysql service start master and slave master server drop db and import from slave to Master db start slave stop slave mysql service stop master and slave changes in my.cnf (mysql master and slave ) 0:02:40 mysql service start master and slave phusion passenger restart both app servers remove maintenance page Testing time not included ====================SHELL SCRIPT WILL TAKE CARE END==================== Total Time 0:20:50 ====== If Build fails on Manual failover and fail back ====== If Build fails on Manual failover and fail back Steps Time connect open vpn 0:00:10 login prodapp01 0:00:10 runcap.sh 0:00:10 maintenance page on 0:00:50 mysql sync break 0:00:50 deploy build on app server 0:05:00 deploy build on collab server 0:11:00 rake task not included (if build fails) stop slave 0:00:20 mysql service stop master and slave 0:00:10 changes in my.cnf (mysql) 0:02:00 mysql service start master and slave 0:00:10 master server drop db and import from slave to Master db 0:03:00 start slave 0:00:10 stop slave 0:00:10 mysql service stop master and slave 0:00:10 changes in my.cnf (mysql master and slave ) 0:02:00 mysql service start master and slave 0:00:10 phusion passenger restart both app servers 0:02:00 remove maintenance page 0:00:50 Testing time not included Total Time 0:30:00