Hello,
Last week my MySQL server was powered off abruptly ( power cut ). When it came back on all the slaves said :
mysqld[6950]: 100909 16:08:37 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position ( server_errno=1236)
mysqld[6950]: 100909 16:08:37 [ERROR] Got fatal error 1236: 'Client requested master to start replication from impossible position' from master when reading data from binary log
So I took a full mysqldump from the master and rebuilt the slaves. The slaves are now synchronized ( according to show slave status\G ( abridged ) ) :
Slave_IO_State: Waiting for master to send event
Seconds_Behind_Master: 0
But I have noticed that the databases are not identical. In one table, I am noticing that about 25% of the write requests from the master are not being run, so I investigated.
I ran mysqlbinlog on all the bin logs on the server. I found that certain transactions are missing in the bin logs but were executed on the master. How could this happen ? All my tables ( almost, mysql.* is still myisam ) are innodb.
So, for some reason, ever since the server sudden power outage, the mysql binlogs are now being written incomplete.
The database resides on a separate partition.
df -h reports
Total : 70G
Used : 29G
Avail : 49G
Use % : 42%
Part of the usage is from backups before they are transfered to a remote machine.
/etc/fstab ( mysql partition )
# /dev/mapper/vg0-mysqldata
UUID=000000000-0000-0000-0000-000000000000 /mnt/mysql xfs relatime 0 2
Server version: 5.0.51a-3ubuntu5.7-log (Ubuntu)
uname -a
Linux db-master 2.6.24-26-server #1 SMP Tue Dec 1 18:26:43 UTC 2009 x86_64 GNU/Linux
It is running on Ubuntu 8.04
In my.cnf
Everything is mostly default settings. There are a few parameters to tweak the innodb performance for our system.
The my.cnf is 235 lines, so please let me know what parts would help understand my issue.
I can't find any relevant messages in the logs. The only message I see is talking about "Aborting connections" because of timeouts, but that's normal for my configuration.
Any ideas?!?!
Thanks!
David
Last week my MySQL server was powered off abruptly ( power cut ). When it came back on all the slaves said :
mysqld[6950]: 100909 16:08:37 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position ( server_errno=1236)
mysqld[6950]: 100909 16:08:37 [ERROR] Got fatal error 1236: 'Client requested master to start replication from impossible position' from master when reading data from binary log
So I took a full mysqldump from the master and rebuilt the slaves. The slaves are now synchronized ( according to show slave status\G ( abridged ) ) :
Slave_IO_State: Waiting for master to send event
Seconds_Behind_Master: 0
But I have noticed that the databases are not identical. In one table, I am noticing that about 25% of the write requests from the master are not being run, so I investigated.
I ran mysqlbinlog on all the bin logs on the server. I found that certain transactions are missing in the bin logs but were executed on the master. How could this happen ? All my tables ( almost, mysql.* is still myisam ) are innodb.
So, for some reason, ever since the server sudden power outage, the mysql binlogs are now being written incomplete.
The database resides on a separate partition.
df -h reports
Total : 70G
Used : 29G
Avail : 49G
Use % : 42%
Part of the usage is from backups before they are transfered to a remote machine.
/etc/fstab ( mysql partition )
# /dev/mapper/vg0-mysqldata
UUID=000000000-0000-0000-0000-000000000000 /mnt/mysql xfs relatime 0 2
Server version: 5.0.51a-3ubuntu5.7-log (Ubuntu)
uname -a
Linux db-master 2.6.24-26-server #1 SMP Tue Dec 1 18:26:43 UTC 2009 x86_64 GNU/Linux
It is running on Ubuntu 8.04
In my.cnf
Everything is mostly default settings. There are a few parameters to tweak the innodb performance for our system.
The my.cnf is 235 lines, so please let me know what parts would help understand my issue.
I can't find any relevant messages in the logs. The only message I see is talking about "Aborting connections" because of timeouts, but that's normal for my configuration.
Any ideas?!?!
Thanks!
David
