I have 3 dedicated servers at OVH, each with i7 4790K, 32GB RAM and 200GB storage
I have in total 10 application on those 3 dedicated, which uses 14 databases. 1 database is on ndbcluster engine and 2 more tables from 2 different databases.
At 5am every day, all 10 applications are stopped and `mysqldump` is called on every database, after dumping is done, 10 applications starts again.
mysqldump command: mysqldump -u backup -pXXXX db_name > db_name.sql
One month ago I setuped MySQL cluster and imported data to it, but I'm facing this problem for every 3 to 4 days. I installed 7.6.11 MySQL cluster, when I run out of ideas what might cause the problem I decided to update to 7.6.12 but problem is still here.
Problem: After applications are stopped, mysqldump do not dump data, it's just wait for timeout and leave empty .sql files, then 10 applications are started and none of them can access mysql API (before they are stopped everything works great), no matter is it NDB tables or InnoDB or MyISAM everything is out of reach, even thou that everything is up and running (ndb_mgmd, ndbd and mysqld). I can login to MySQL with phpmyadmin but I can not select any data base it just stuck on "Loading"
Solution: I get into managments server (ndb_mgm) and restart !one! of data nodes ("5 restart" or "5 restart") and then mysqldump can dump the data from database. Also my 10 application can access mysql API servers.
I have not noticed data loss.
Every time this happens I checked logs, at first I noticed that some drive is full, later I release that "tar" (linux command) filled storage device while backuping files and directories from 10 application that I have. I solved it and it do not happens again, but the mysql problem is still here. Now I do not see any problem in logs.
Since OVH gives only 20GB for root device I moved temp directory that MySQL uses to /home directory (which have 200GB) but it did not solved the problem.
I collected logs with "ndb_error_reporter" but I need to keep my IPs private so I replaced them with local IPs
Link for logs: https://mega.nz/#!uEsWyIbY!zF8IuglWtN-7mRttiABapBJk6jnL007HI-j2muwKUk0
What is causing this issue?