Re: Power outage db down - crisis over

[Bob Bankay]

Heikki,

Thanks for your several replies.

At one point last week we decided to restore the Opteron master from the slave Sparc. Mysqld was getting lost every time it tried to do anything with the ibdata files on the master. So we re-created the ibdata and log files.

The data migration went as well as could be expected. We lost about 30 minutes worth of processing (thank goodness we are dealing with scientific data and not cash). The lost data will be reprocessed and we have invested in intelligent UPSs which can trigger automatic shutdown scripts. We expect to have 5 minutes survivabilty and are setting our innodb dirty_buffer_pct to make it around 105K pages(45%). This can be flushed in about 3 minutes at shutdown, so it should provide enough margin. Right now this limit will not impact performance.

This week we were able to prove the worth of our work since we had another power outage 36 hours ago. And this time there was no crisis or data loss. Lesson learnt.

PS I was interested in the comment about mounting the disks in Linux, setting hdparam to not cache. I will verify with my admin guy that disk mounting is set for no caching in the Opteron.

Thanks,
Bob Bankay
SETI@Home