Data loss after some days of migration
I'm writing there to find an answer of what had append to me...
At 19 of July i was running a server (server1) with ispconfig and 2-3 website with mysql databases, all was working fine.
In that date i've decided to migrate to a new server (server2), to do that i've done a fresh install of ispconfig on the new server, to migrate the databases, i've done as follow.
- stopped mysql on server1
- stopped mysql on server2
- copied /var/lib/mysql from server1 to /var/lib/mysql to server2
- started mysql on server2 (copied all file of ispconfig and changed the various password around)
- resynced all servrices
from my point of view all was working fine, all db where up ad running and all website were working ok... but at this point become the problems...
The customer called me, saying that some romecord on database1 where missing (strange thing... all other things where ok...), so for this particular database i've decided to reimport data with mysqldump.
At this point all data where imported correctly, and the website where up and running. From this point costumer started working on site without any issue.
Yesterday we got and we had to restart the virtualization node with all the vm on it, included the one i'm talking about.. and after the reboot, come the trick!!
All other virtual machine and databases were ok, but after restart the database1 on server2 presented me alla data of 19 July instead of data updated to 31 July (WTF!!!), at this point i thought db was corrupted and infact with a repair some table were fixed, losing some record abviously.
At this point i've decided to come back with a backup fo the virtual machine of the previous day and here another strange thing... alla consistent data were accessible only from 19 July, but people working on site had updated the database correctly until this point.
I've check all the backup of the previous day until 28 July, all the same problems.
In the meanwhile customer hasn't checked if backup inside mysql was working, and he noticed only yesterday (customaer fail...) that backup was stopped at 19 July (i've investegated and fixed the issue and not related to mysqldump or similar but other things, so essentially, backup never started.
I've checked log for errors, for any issue but nothing found....
The question is simply: what can be happened to mysql in this situation?
Both servers where running the same version of Mysql 5.5 on Debian 8