Massive increase in InnoDB corruption issues since upgrade to 8.0.29

[Ralf Meyer]

Dear MySQL Forums Community,


we have recently (about three weeks ago) upgraded our MySQL 8 servers to the current version 8.0.29 from 8.0.25 / 8.0.26.

Since then, we are seeing frequent InnoDB data corruption, which in most cases forces us to restore from backups. In these cases, the server in question usually crashes with an InnoDB Assertion failure / corruption error message right at startup and goes into a restart loop until we stop it using systemctl. That obviously causes some data loss, as the server usually will not start successfully anymore and thereby quite a bit of work for us. To put this into perspective: We are currently running around 300 MySQL 8 servers. Starting this week we so far had 6 servers which had InnoDB corruption. So 6 servers within 7 days (out of approx. 300). Usually we might have one case of InnoDB corruption which causes crashes once per year or so, if even that.

On four servers, we had to restore from backups, as innodb_force_recovery until level 3 would not allow us to start the database again.
On one server, the server crashed randomly every few minutes, which at least allowed us to save current dumps and then reinitialize the server and restore them (so we did not have to resort to backups).
On the sixth server, we were able to recover with innodb_force_recovery=2, dumping the database table in question and reimporting it.

We have saved the corrupted data dir from all these systems for analysis purposes. It seems that a start with innodb_force_recovery 6 would be theoretically possible in all cases, however due to the risk of massive data corruption, so far we prefer to restore from backups, if a start with a value of 3 or lower is not possible. That being said, even the use of innodb_force_recovery at a low level of 1 - 3 was not a usual occurance for us so far.

There does not seem to be an exact problem that we can pinpoint, as is often the issue with these kinds of data corruptions. In some cases, another InnoDB corruption crash (followed by a successful restart, though) occured a few days before the start preventing corruption happened, but this is also not consistent on all systems. The error messages output by MySQL differ quite a bit per system, so I am not sure, if it even makes any sense to provide them all here. All these systems are running on different KVM hosts and we did not have these problems before upgrading to 8.0.29, which is why I guess it might be related to that upgrade. Since we came from 8.0.25 / 8.0.26, it could potentially also be related to changes made in versions 8.0.27 / 8.0.28. The servers are used by many users in many different, individual ways, so we unfortunately cannot relate the problems to any specific application so far.

I was wondering if maybe some other users are experiencing the same and it just is not being talked about online so far (which is why I am creating this thread). I would also appreciate any tips on what we could try to pinpoint this issue any further, as we are not sure how to debug this any further and are worried about more servers crashing and needing to be restored by us, as this behaviour is something we have never seen before, especially in this short timeframe over many different servers on different KVM host systems. I think that the information that we have so far might not be sufficient for creating a bug report that actually leads to some kind of resolution.

Thank you for your help.