Programming Tutorials

This post is the fourth of a series that started here.
From the previous of this series, we now have resources configured but instead of starting MySQL, Pacemaker invokes a script to start (or restart) the EC2 instance running MySQL. This blog post describes the instance restart script. Remember, I am more a DBA than a script writer so it might not be written in the most optimal way.
First, let's recap what's the script has to perform (the full script is given below).

1. Kill the MySQL EC2 instance if running
2. Make sure the MySQL EC2 instance is stopped
3. Prepare the user-data script for the new MySQL EC2 instance
4. Launch the new MySQL instance
5. Make sure it is running
6. Reconfigure local heartbeat
7. Broadcast the new MySQL instance IP to the application servers

Kill the MySQL EC2 instance
In order to kill the existing MySQL EC2 instance, we first have to identify it. This is done by:
PLAIN TEXT
CODE:

1. OLD_INSTANCE_ID=`ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $AMI_HA_MYSQL | egrep "running|pending" | tail -n 1 | cut -d'|' -f3`

by filtering on the AMI type of the instance. Since an instance can be listed at the "stopped" state, it is mandatory to filter for states "running" or "pending". Then the instance is terminated with:
PLAIN TEXT
CODE:

1. ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null

Make sure the MySQL EC2 instance is stopped
Terminating an EC2 instance is not instantaneous, we can confirm an instance is really stopped by monitoring its status and wait until it is actually "terminated". The code below is how the script performs this task.
PLAIN TEXT
CODE:

1. #wait until the old instance is terminated  it takes a few seconds to stop
2. done="false"
3. while [ $done == "false" ]
4. do
5.     status=`ec2-describe-instances -K $PK -C $CERT $OLD_INSTANCE_ID | /usr/local/bin/filtre_instances.pl |  grep -c terminated`
6.       if [ "$status" -eq "1" ]; then
7.             done="true"
8.         else
9.                 ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
10.             sleep 5
11.      fi
12. done

Prepare the user-data script for the new MySQL EC2 instance
The new MySQL instance will be running heartbeat. Since we cannot use neither Ethernet broadcast or multicast, we need to configure the new instance so that it communicates through unicast with its partner node in the cluster, the node on which the restart script is run. This configuration is achieved by providing a user-data script (see the hamysql.user-data below) which completes the heartbeat configuration of the new instance. The hamysql.user-data script only performs a search and replace operation on the /etc/ha.d/ha.cf file and then restart the heartbeat service. In order for this to work properly, we just have to put the IP of the current instance in the script like here:
PLAIN TEXT
CODE:

1. OUR_IP=`/sbin/ifconfig eth0 | grep 'inet addr' | cut -d':' -f2 | cut -d' ' -f1`
2. #Now, modify the user-data script, we need to put our IP address in
3. if [ "$OUR_IP" == "" ]
4. then
5.     echo "Error getting Our IP"
6. else   
7.     perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $OUR_IP/g" $USER_DATA_SCRIPT
8. fi

Launch the new MySQL instance
Once things are ready, a new MySQL instance can be launched with:
PLAIN TEXT
CODE:

1. #Now we are ready to start a new one
2. INSTANCE_INFO=`ec2-run-instances -K $PK -C $CERT $AMI_HA_MYSQL -n 1 -g $HA_SECURITY_GROUP -f $USER_DATA_SCRIPT -t $INSTANCE_TYPE -z $INSTANCE_ZONE -k $INSTANCE_KEY | /usr/local/bin/filtre_instances.pl`
3.  
4. #wait until the new instance is running  it take a few seconds to start
5. NEW_INSTANCE_ID=`echo $INSTANCE_INFO | cut -d'|' -f3`

Out of this operation, we retrieve the new instance "instance_id".
Make sure it is running
Since we know the "instance_id" of the new instance, checking if it is running is easy:
PLAIN TEXT
CODE:

1. done="false"
2. while [ $done == "false" ]
3. do
4.     INSTANCE_INFO=`ec2-describe-instances -K $PK -C $CERT  $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl`
5.     status=`echo $INSTANCE_INFO | grep -c running`
6.     if [ "$status" -eq "1" ]; then
7.         done="true"
8.     else
9.         sleep 5
10.     fi
11. done

Reconfigure local heartbeat
Now, Heartbeat, on the monitoring host, must be informed of the IP address of its new partner. In order to achieve this, a search and replace operation in the local ha.cf file followed of restart of Heartbeat is sufficient.
PLAIN TEXT
CODE:

1. #Set the IP in /etc/ha.d/ha.cf and ask heartbeat to reload its config
2. MYSQL_IP=`ec2-describe-instances -K $PK -C $CERT  $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | cut -d'|' -f2`
3. perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $MYSQL_IP/g" /etc/ha.d/ha.cf
4. /etc/init.d/heartbeat reload

Broadcast the new MySQL instance IP to the application servers
The final phase is to inform the application servers that the IP of the MySQL has changed. The best way to list those application servers is through a security group and, provided the appropriate ssh keys have been exchanged, this code will push the IP update.
PLAIN TEXT
CODE:

1. TMPFILE=`mktemp`
2. ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $CLIENT_SECURITY_GROUP> $TMPFILE
3.    
4. while read line
5. do
6.     IP=`echo $line | cut -d'|' -f2`
7.     ssh -i /usr/local/bin/update_mysql ubuntu@$IP sudo ./updated_xinetd.sh $MYSQL_IP
8. done <$TMPFILE
9.  
10. rm $TMPFILE

The full script:
PLAIN TEXT
CODE:

1. #!/bin/bash
2. HA_SECURITY_GROUP=testyves
3. CLIENT_SECURITY_GROUP=hamysql-client
4. CLIENT_SCRIPT=/usr/local/bin/update_client.sh
5. AMI_HA_MYSQL=ami-84a74fed
6. EBS_DATA_VOL=vol-aefawf
7. USER_DATA_SCRIPT=/usr/local/bin/hamysql.user-data
8. PK=/usr/local/bin/pk-FNMBRRABFRKVICBDZ4IOOSF7YROYZRZW.pem
9. CERT=/usr/local/bin/cert-FNMBRRABFRKVICBDZ4IOOSF7YROYZRZW.pem
10. INSTANCE_TYPE=m1.small
11. INSTANCE_ZONE=us-east-1c
12. INSTANCE_KEY=yves-key
13.  
14. OUR_IP=`/sbin/ifconfig eth0 | grep 'inet addr' | cut -d':' -f2 | cut -d' ' -f1`
15. #Now, modify the user-data script, we need to put our IP address in
16. if [ "$OUR_IP" == "" ]
17. then
18.     echo "Error getting Our IP"
19. else   
20.     perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $OUR_IP/g" $USER_DATA_SCRIPT
21. fi
22.  
23. while [ 1 ]; do
24.  
25.     #First thing to do, terminate the other instance ID
26.     OLD_INSTANCE_ID=`ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $AMI_HA_MYSQL | egrep "running|pending" | tail -n 1 | cut -d'|' -f3`
27.  
28.     if [ "$OLD_INSTANCE_ID" == "" ]
29.     then
30.         #no running instance
31.         :
32.     else
33.         ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
34.  
35.         #wait until the old instance is terminated  it takes a few seconds to stop
36.         done="false"
37.         while [ $done == "false" ]
38.         do
39.             status=`ec2-describe-instances -K $PK -C $CERT $OLD_INSTANCE_ID | /usr/local/bin/filtre_instances.pl |  grep -c terminated`
40.          if [ "$status" -eq "1" ]; then
41.               done="true"
42.       else
43.                 ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
44.                 sleep 5
45.       fi
46.         done   
47.     fi
48.  
49.     #Now we are ready to start a new one
50.     INSTANCE_INFO=`ec2-run-instances -K $PK -C $CERT $AMI_HA_MYSQL -n 1 -g $HA_SECURITY_GROUP -f $USER_DATA_SCRIPT -t $INSTANCE_TYPE -z $INSTANCE_ZONE -k $INSTANCE_KEY | /usr/local/bin/filtre_instances.pl`
51.  
52.     #wait until the new instance is running  it take a few seconds to start
53.     NEW_INSTANCE_ID=`echo $INSTANCE_INFO | cut -d'|' -f3`
54.  
55.     if [ "$NEW_INSTANCE_ID" == "" ]
56.     then
57.         echo "Error creating the new instance"
58.     else
59.    
60.         done="false"
61.         while [ $done == "false" ]
62.         do
63.       INSTANCE_INFO=`ec2-describe-instances -K $PK -C $CERT  $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl`
64.          status=`echo $INSTANCE_INFO | grep -c running`
65.       if [ "$status" -eq "1" ]; then
66.               done="true"
67.          else
68.                 sleep 5
69.       fi
70.         done
71.  
72.         #Set the IP in /etc/ha.d/ha.cf and ask heartbeat to reload its config
73.         MYSQL_IP=`ec2-describe-instances -K $PK -C $CERT  $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | cut -d'|' -f2`
74.         perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $MYSQL_IP/g" /etc/ha.d/ha.cf
75.  
76.         TMPFILE=`mktemp`
77.         ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $CLIENT_SECURITY_GROUP> $TMPFILE
78.    
79.         while read line
80.         do
81.             IP=`echo $line | cut -d'|' -f2`
82.             ssh -i /usr/local/bin/update_mysql ubuntu@$IP sudo ./updated_xinetd.sh $MYSQL_IP
83.         done <$TMPFILE
84.  
85.         rm $TMPFILE
86.        
87.         /etc/init.d/heartbeat reload
88.     fi
89.  
90.  
91.         sleep 300 # 5 min before attempting again. Normally heartbeat should kill the script before
92. done

The hamysql.user-data script:
The script sets the IP of the monitor host in the heartbeat ha.cf configuration file and then, finishes up some missing configuration settings of the AMI.
PLAIN TEXT
CODE:

1. #!/bin/bash
2. sudo hostname hamysql
3. sudo perl -pi -e "s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 10.220.230.18/g" /etc/ha.d/ha.cf
4.  
5. # to eventually be added to the ebs image
6. sudo perl -pi -e 's/bind-address/#bind-address/g' /etc/mysql/my.cnf
7. sudo service mysql restart
8. sleep 5
9. /usr/bin/mysql -u root -proot -e "grant all on *.* to root@'%' identified by 'root'"
10. sudo /etc/init.d/heartbeat start

Entry posted by Yves Trudeau |
No comment
Add to: | | | |