Question about Innodb Cluster Failure condition

[Junaid Zeeshan]

I am running MySQL innodb cluster with multi master group replication + MySQL router to distribute traffic. I want to know at which condition cluster will stop work and we can face outage. I have 5 nodes cluster in my dev environment and according to documentation if half number of the nodes will be down cluster will not work.
I tried to produce this failed scenario but couldn't succeed. I used multiple ways to down 3 nodes out of 5 but cluster still works for read and write through mysql router. I stopped group replication, killed mysql service or rebooted 3 servers but cluster still works.
Can you please guide, how can we test this failed scenario of cluster when more than half number of nodes are down.

Secondly, for production we have two data centers and we have plan to deploy four nodes in each data center. So if one data center is down, we still have four nodes, but according to documentation if 50% nodes are down, cluster should not work. Please suggest if we should deploy 9th node in AWS to make cluster up if one of the data center is down.

Looking forward your earliest response as have short time to prepare architecture of our production environment. thank you



Edited 1 time(s). Last edit at 12/05/2023 12:42AM by Junaid Zeeshan.