Time steps for deployment
step 1 00:00:10 Vpn connect and login to prodapp01
step 2 00:00:50 maintenance page on
step 3 00:00:30 mysql sync break
step 4 Runcap.sh build deployment .(back fill tasks)
if build fails step 5 to step 8 applicable
step 5 00:02:00 database.yml fie change both app01 and app02 servers replace mysql master ip to slave ip
step 6 00:00:05 stop mysql service slave server
step 7 00:02:00 my.cnf change slave db as master server and start mysql server
step 8 00:05:00 both app server restart passenger service
step 9 00:00:40 remove maintenance page and site able to access
11.25 Total Time
steps for rollback with mysqldump (Fail Back) option1
step 1 00:00:10 Vpn connect
step 2 00:00:10 Drop database in mysql master
step 3 00:03:00 create new db and import mysql dump
step 4 Once mysql db identical master and slave (Based on db size time differ)
step 5 00:03:00 swap my.cnf file in mysql master and slave servers and restart mysql service both servers
step 6 00:00:10 login to prodapp01
step 7 00:00:50 maintenance page on
step 8 00:05:00 database.yml fie change both app01 and app02 servers db masterip change
step 9 00:00:40 both app server restart passenger service
step 10 00:00:50 remove maintenance page
13.83 Total Time
#!/bin/bash #title : mysql failover and failback #author : Sudharsan #Designation : Devops #10.166.152.15 MYSQL MASTER #10.166.152.23 MYSQL SLAVE #path /home/expprodl/bin/convert_as_master.sh mysql -u repl -p******** -h10.166.152.15 -P 3306 -e "STOP SLAVE;" mysql -u repl -p******** -h10.166.152.15 -P 3306 -e "RESET MASTER;" mysql -u repl -p******** -h10.166.152.23 -P 3306 -e "STOP SLAVE;" mysql -u repl -p******** -h10.166.152.23 -P 3306 -e "RESET MASTER;" ssh root@10.166.152.23 "/etc/init.d/mysqld stop" ssh root@10.166.152.15 "/etc/init.d/mysqld stop" echo "my.cnf changes done 10.166.152.15 MYSQL MASTER " ssh root@10.166.152.15 "cp /etc/.my.cnf-master /etc/my.cnf" #10.166.152.15 converted to master echo "my.cnf changes done 10.166.152.23 MYSQL SLAVE " ssh root@10.166.152.23 "cp /etc/.my.cnf-slave /etc/my.cnf" #10.166.152.23 converted to Slave ssh root@10.166.152.23 "/etc/init.d/mysqld start" ssh root@10.166.152.15 "/etc/init.d/mysqld start" echo "restart passenger on 10.166.152.14" ssh expprodl@10.166.152.14 "touch /deploy/crossbow/current/tmp/restart.txt" echo "restart passenger on 10.166.152.22" ssh expprodl@10.166.152.22 "touch /deploy/crossbow/current/tmp/restart.txt" sleep 30 echo "Please wait for a while passenger service restarting"
#!/bin/bash
#title : replication
#description : This script automates the process of starting a Mysql Replication
#author : Sudharsan
#Designation : Devops
#Date : 31-Jan-2017
#path /home/expprodl/bin/replication.sh
DB=cbprod
DUMP_FILE="/tmp/$DB-export-$(date +"%Y%m%d%H%M%S").sql"
USER=repl
PASS=********
MASTER_HOST=10.166.152.15
SLAVE_HOSTS=(10.166.152.23)
echo "MASTER: $MASTER_HOST"
mysql -h $MASTER_HOST "-u$USER" "-p$PASS" $DB <<-EOSQL &
GRANT REPLICATION SLAVE ON *.* TO '$USER'@'%' IDENTIFIED BY '$PASS';
FLUSH PRIVILEGES;
FLUSH TABLES WITH READ LOCK;
DO SLEEP(3600);
EOSQL
echo " - Waiting for database to be locked"
sleep 3
echo " - Dumping database to $DUMP_FILE"
mysqldump -h $MASTER_HOST "-u$USER" "-p$PASS" --opt $DB > $DUMP_FILE
echo " - Dump complete."
MASTER_STATUS=$(mysql -h $MASTER_HOST "-u$USER" "-p$PASS" -ANe "SHOW MASTER STATUS;" | awk '{print $1 " " $2}')
LOG_FILE=$(echo $MASTER_STATUS | cut -f1 -d ' ')
LOG_POS=$(echo $MASTER_STATUS | cut -f2 -d ' ')
echo " - Current log file is $LOG_FILE and log position is $LOG_POS"
kill $! 2>/dev/null
wait $! 2>/dev/null
echo " - Master database unlocked"
for SLAVE_HOST in "${SLAVE_HOSTS[@]}"
do
echo "SLAVE: $SLAVE_HOST"
echo " - Creating database copy"
mysql -h $SLAVE_HOST "-u$USER" "-p$PASS" -e "DROP DATABASE IF EXISTS $DB; CREATE DATABASE $DB;"
scp $DUMP_FILE $SLAVE_HOST:$DUMP_FILE >/dev/null
mysql -h $SLAVE_HOST "-u$USER" "-p$PASS" $DB < $DUMP_FILE
echo " - Setting up slave replication"
mysql -h $SLAVE_HOST "-u$USER" "-p$PASS" $DB <<-EOSQL &
STOP SLAVE;
CHANGE MASTER TO MASTER_HOST='$MASTER_HOST',
MASTER_USER='$USER',
MASTER_PASSWORD='$USER',
MASTER_LOG_FILE='$LOG_FILE',
MASTER_LOG_POS=$LOG_POS;
START SLAVE;
EOSQL
# Wait for slave to get started and have the correct status
sleep 2
# Check if replication status is OK
SLAVE_OK=$(mysql -h $SLAVE_HOST "-u$USER" "-p$PASS" -e "SHOW SLAVE STATUS\G;" | grep 'Waiting for master')
if [ -z "$SLAVE_OK" ]; then
echo " - Error ! Wrong slave IO state."
else
echo " - Slave IO state OK"
fi
done
If Build fails on shell script failover and fail back Steps Time connect open vpn 0:00:10 login prodapp01 0:00:10 runcap.sh 0:00:10 maintenance page on 0:00:50 mysql sync break 0:00:50 deploy build on app server 0:05:00 deploy build on collab server 0:11:00 rake task not included (if build fails) ====================SHELL SCRIPT WILL TAKE CARE START==================== stop slave mysql service stop master and slave changes in my.cnf (mysql) mysql service start master and slave master server drop db and import from slave to Master db start slave stop slave mysql service stop master and slave changes in my.cnf (mysql master and slave ) 0:02:40 mysql service start master and slave phusion passenger restart both app servers remove maintenance page Testing time not included ====================SHELL SCRIPT WILL TAKE CARE END==================== Total Time 0:20:50
If Build fails on Manual failover and fail back Steps Time connect open vpn 0:00:10 login prodapp01 0:00:10 runcap.sh 0:00:10 maintenance page on 0:00:50 mysql sync break 0:00:50 deploy build on app server 0:05:00 deploy build on collab server 0:11:00 rake task not included (if build fails) stop slave 0:00:20 mysql service stop master and slave 0:00:10 changes in my.cnf (mysql) 0:02:00 mysql service start master and slave 0:00:10 master server drop db and import from slave to Master db 0:03:00 start slave 0:00:10 stop slave 0:00:10 mysql service stop master and slave 0:00:10 changes in my.cnf (mysql master and slave ) 0:02:00 mysql service start master and slave 0:00:10 phusion passenger restart both app servers 0:02:00 remove maintenance page 0:00:50 Testing time not included Total Time 0:30:00