Re: Any benefit to have more than 1 mgm running?
Hi,
In the normal case, even if you lose the mgm and a data node, you still will have a running cluster. However if the case where you have just one other data node, then the entire cluster would shutdown due to losing a majority of the data nodes with no arbitrator to resolve it.
Assuming that you have more than 2 data nodes (3,4,5, etc...) then the cluster should be able to continue running, even if the mgm and a data node goes down. The mgm however would be required to be restarted before the data node could rejoin.
On thing to note is that there is the possibility to have redundant mgm nodes, however it just hasn't been all that throughly tested and widely used due to the fact the failure of the mgm doesn't really have any impact on a running cluster.
The answer to the last question lies in how the mgm node crashed. If the server it is on stops running, as you said above, then you would need to restart the server which would restart the mgm (assuming you have set it up in the init.d). Otherwise it would be trivial to create a script for restart if you so desire. (Based on mysqld_safe even)
Generally for monitoring what status the different nodes are in, the mgm is most commonly used for that. It can say what nodes are down/up/etc... So your monitoring program talks to the mgm directly, and if it goes down, the monitoring program will definately notice.
Regards,
Harrison