Recovery after node restart

[Annie Blomqvist]

Hi,

I'm running a K3S cluster on a number of virtual Debian 6.1.106-3 hosts:

Client Version: v1.29.9
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0+k3s1

Its running on 3 masters and 3 worker nodes and is using metallb for loadbalancing.

I've created a mysql cluster with the helm chart:

helm upgrade --install mysql-cluster mysql-operator/mysql-innodbcluster --namespace test --create-namespace -f mysql-cluster/mysql-cluster-values.yaml

It seems like the router is not able to recover when the worker node is restarted.

kubectl logs -n test pods/mysql-cluster-router-7988469f76-fcbff --all-containers
[Entrypoint] MYSQL_CREATE_ROUTER_USER is 0, Router will reuse mysqlrouter account at runtime
[Entrypoint] Succesfully contacted mysql server at mysql-cluster-instances.test.svc.cluster.local:3306. Checking for cluster state.
[Entrypoint] Succesfully contacted mysql server at mysql-cluster-instances.test.svc.cluster.local. Trying to bootstrap reusing account "mysqlrouter".
Please enter MySQL password for mysqlrouter:
Error: The provided server is currently not an ONLINE member of a InnoDB cluster.

I was finally able to recover the cluster by restarting all worker nodes, after which I could perform a dba.rebootClusterFromCompleteOutage()

I thought by using a cluster and by using the operator I would have a system able to handle single node failure?
Should the operator not be able to survive and recover from a temporary failure of one of the cluster nodes?