Re: Disaster Recovery with master-master replication

[Rick James]

> As there would inevitably be a short period where we inadvertently write to both masters.

Do two things:
* SET read_only = ON; -- on the backup master.
* Do not use 'root' (or other SUPER user) for your applications. (Use only for admin stuff by a human.)

The second item is because SUPER ignores read_only.

Be sure to set read_only at the proper time in your failover scripts. You _will_ have a brief time where no one can write.

DNS is notoriously slow; I suspect EC2 is no better. The stated TTL cannot be trusted.

> too long to re-read all the binary logs
As time goes on, you will get more logs. This will become intractable. Start considering taking dumps and use the logs from the point of the dump.

Another issue. If the disk dies on the writable master, you will need to take the backup master down long enough to take a dump to recreate the Master. Suggest you look into using Xtradb and Xtrabackup -- with them, you can almost avoid any downtime to take a dump of a running Master.