Putting MySQL on hold for backup

[Sam D]

We are running a reasonably large MySQL database (> 100.000 tables, > 4 TB size).
The database is used to store incoming measurement data, which is pushed by measurement stations, so that we do not really have control over when and how much data is coming in.

The database runs under SLES and VMware. The backup is executed as a virtual machine backup, not with database tools. This runs fine for most of the times, but randomly, the backup and large incoming data loads overlap which in turn drives the backup into a desaster: running for days, eating up all disk space, etc. On the other hand, a stop of MySQL at a given time would interrupt a currently running data load, which is for many reasons not very beneficial.

Of course, we could start to write loading software, which is more fault tolerant and digests reloads after being interrupted. We could implement semaphores, which coordinate loading procedures and backup procedures.

An easy way would be just to put the database deamon processes on hold on the OS level, so that they don't write during the backup and release them afterwards.

Any experience with such a procedure ? Any hint for a semaphore framework as an alternative ? Thank you for any comment on these thoughts in advance.
Sam.