Re: Replication & failover with 4 servers
Posted by:
Rick James
Date: September 26, 2010 10:29AM
There are 3rd party products that, for a price, do a pretty good job of automating failover. Tungsten Continuent
I don't trust trying to automate things. If the network between A and B goes down, the monitoring could mistakenly decide that all of A is down, and promote a machine in B to be Master. Then you have two Masters; this would lead to data corruption that would be much too nasty to unravel.
Dual Master has 2 machines replicating from each other. I strongly recommend you actually write to only one of them. This implies that the replication going from the backup master to the live master is actually doing nothing. But it is established, thereby making failover easier (one less thing to do).
But... You can't have Dual Master set up with two different slaves. (A Slave cannot have more than one Master; this would need the live Master to be a Slave to two backup masters.)
Are A and B geographically separated? I am thinking about a flood, earthquake, tornado, etc, wiping out A or B, but not both.
Once the Master dies and you promote some slave, you are left with a nightmare of rebuilding the tree so that the other slaves point to the new master. This is quite difficult. Recommend, instead, that you have one slave hanging off each of the Dual-Master machines. That way, when the live master dies, the backup master plus its slave can take over while you sort out what to do with the slave hanging off the dead master.
Look at the "Google Patch". It provides a reasonably clean way to have a single Master, plus an arbitrary tree, and failover efficiently to the "best" slave.