Tuesday, November 26, 2019

Repair GTID Based Slave on Percona Cluster


Problem : 

We are running 5 node percona cluster on Ubuntu 16.04, and its configured with master-slave replication. Suddenly we got an alert for replica broken from slave server, which was earlier configured with normal replication 


We have tried to sync the data and configure the replication, unable to fix that immediately due to huge transactions and GTID enabled servers. So we have decided to follow with innobackupex tool, and problem fixed in 2 hours 

Followed all the steps from percona doc and shared the experience in my environment 

Steps involving to repair the broken Replication :

1.Backup master server 
2.Prepare the backup 
3.Restore and Configure the Replication
4Check Replication Status

1.Backup master server 

We need to configure the complete master server database into a slave. So we are taking a full backup from the master server, before proceeding the backup we should check disk space available for the backup because of its system-level backup 



We have created a specific user for taking a backup from master server, once the backup is completed will get OK like below,


2. Prepare the backup for Restore

We need to prepare the backups to apply the transaction logs into data files, once it is OK, data files are ready to restore 



Before moving the prepared files into slave server, verify the GTID information from xtrabackup_binlog_info 


3.Restore and Configure the Replication

We can restore the backup else to create a new data directory and move the files into the directory. We have followed created a new directory and change datadir values in mysqld.conf file.

Once changed the data directory, we need to change the owner and permission of the MySQL data directory 

chown mysql:mysql /mnt/mysqldatanew
And restart the service with a new data directory, once its started login with master MySQL root user password. Because we have taken file backup from the master so metadata will be the same of master
Execute below commands to configure the replication 
4. Check Replication Status

Once slave is configured, verify the replication status as below 


Also, Slave has retrieved a new transaction 


Thanks for Reading !!!

No comments: