OS: Ubuntu 16.04.03 x64
MySQL: 5.7.23-community (MySQL ppa repo)
Anyone else running Ubuntu 16.04.3 with kernel 4.15.0-36-generic and having sluggish replication performance?
We have a MySQL farm of seven severs, all running Ubuntu on dedicated hardware. Ubuntu released kernel 4.15.0-36-generic last week and we rebooted this last weekend. Replication has been falling steadily behind since the reboot. There was no reason for it that we could see. The servers were not choaking under any load; network, storage, or CPU. Servers on the same network switch were up to hours behind in replication. This was something new. We have never really had a significant lag before - usually a minute or so was the worst we ever had. But hours?
There were no MySQL or OS configurations changes. All servers were updated to MySQL 5.7.23 on 2018-08-08 and have been running fine since that update.
The biggest indicator of the lag was "SLAVE RETRIEVED". It was way behind. "SLAVE EXECUTED" was keeping up but "SLAVE RETRIEVED" seem to not be downloading transactions from the upstream host with any urgency.
Long story short, we eliminated the physical network and concluded it had to be an OS update. We restarted one server using kernel 4.15.0-34-generic and the server caught up replication in just a couple of minutes!
MySQL: 5.7.23-community (MySQL ppa repo)
Anyone else running Ubuntu 16.04.3 with kernel 4.15.0-36-generic and having sluggish replication performance?
We have a MySQL farm of seven severs, all running Ubuntu on dedicated hardware. Ubuntu released kernel 4.15.0-36-generic last week and we rebooted this last weekend. Replication has been falling steadily behind since the reboot. There was no reason for it that we could see. The servers were not choaking under any load; network, storage, or CPU. Servers on the same network switch were up to hours behind in replication. This was something new. We have never really had a significant lag before - usually a minute or so was the worst we ever had. But hours?
There were no MySQL or OS configurations changes. All servers were updated to MySQL 5.7.23 on 2018-08-08 and have been running fine since that update.
The biggest indicator of the lag was "SLAVE RETRIEVED". It was way behind. "SLAVE EXECUTED" was keeping up but "SLAVE RETRIEVED" seem to not be downloading transactions from the upstream host with any urgency.
Long story short, we eliminated the physical network and concluded it had to be an OS update. We restarted one server using kernel 4.15.0-34-generic and the server caught up replication in just a couple of minutes!