Hi,
We have on our production environment 2 machines that are set up to do selective replication. The master is a Linux running MySQL 5.0.77 and slave is 5.1.51 running on a Win2008 R2.
A few days ago we added an additional set of tables to be replicated on a second DB between these instances and thus restarted master and slave.
After this change replication began to work very slow. So slow in fact that slave has been opening a gap in Seconds_Behind_Master since the change took place. We tried several more config changes and restarts that did not help.
Relevant stats:
Slave_Io_State: Waiting for master to send event
Master_Log_file: mysql-bin.000408 (which is verified as physically the latest file on master)
relay_log_file: mysql-relay-bin.000047 (physically on slave disk we have up to mysql-relay-bin.000143)
relay_master_log_file: mysql-bin.000380 (as I said we're lagging)
seconds_behind_master: 127889
Show processlist gives:
+ ------- + --------- + --------- + ------- + ------------ + --------- + ---------- + --------- +
| Id | User | Host | db | Command | Time | State | Info |
+ ------- + --------- + --------- + ------- + ------------ + --------- + ---------- + --------- +
| 1 | system user | | | Connect | 68039 | Waiting for master to send event | |
| 2 | system user | | | Connect | 128345 | Reading event from the relay log | |
So it seems like the copying of the binlog files from master works just fine and it's just the read/execution on slave itself that's lagging.
Here's what I checked on slave:
1. As I said a restart for slave was attempted
2. no disk space issue for the MySQL partition is observed
3. No high CPU or low memory is observed on slave box
4. HOWEVER, in correlation to when the issues started I see a rise in disk Queue length. where previously it was sinusoidal looking on a daily cycle, now it just rose (above normal levels) and remains constant that way.
I am assuming note 4 is directly related to the problem, but I am not sure how to proceed.
Any suggestions what my next step should be?
Thanks,
DB
We have on our production environment 2 machines that are set up to do selective replication. The master is a Linux running MySQL 5.0.77 and slave is 5.1.51 running on a Win2008 R2.
A few days ago we added an additional set of tables to be replicated on a second DB between these instances and thus restarted master and slave.
After this change replication began to work very slow. So slow in fact that slave has been opening a gap in Seconds_Behind_Master since the change took place. We tried several more config changes and restarts that did not help.
Relevant stats:
Slave_Io_State: Waiting for master to send event
Master_Log_file: mysql-bin.000408 (which is verified as physically the latest file on master)
relay_log_file: mysql-relay-bin.000047 (physically on slave disk we have up to mysql-relay-bin.000143)
relay_master_log_file: mysql-bin.000380 (as I said we're lagging)
seconds_behind_master: 127889
Show processlist gives:
+ ------- + --------- + --------- + ------- + ------------ + --------- + ---------- + --------- +
| Id | User | Host | db | Command | Time | State | Info |
+ ------- + --------- + --------- + ------- + ------------ + --------- + ---------- + --------- +
| 1 | system user | | | Connect | 68039 | Waiting for master to send event | |
| 2 | system user | | | Connect | 128345 | Reading event from the relay log | |
So it seems like the copying of the binlog files from master works just fine and it's just the read/execution on slave itself that's lagging.
Here's what I checked on slave:
1. As I said a restart for slave was attempted
2. no disk space issue for the MySQL partition is observed
3. No high CPU or low memory is observed on slave box
4. HOWEVER, in correlation to when the issues started I see a rise in disk Queue length. where previously it was sinusoidal looking on a daily cycle, now it just rose (above normal levels) and remains constant that way.
I am assuming note 4 is directly related to the problem, but I am not sure how to proceed.
Any suggestions what my next step should be?
Thanks,
DB