MySQL Group Replication Won't start

[Tim Igoe]

Here's a puzzle... I have a 2 server group replication setup, I had to restart the cluster to change some settings, and since then I have been unable to get it to start back up. Both instances of MySQL are running, but the cluster will not bootstrap anymore.

Taking one of the servers in the cluster, when starting MySQL it sits in read only mode for a period before eventually giving up with the group replicaiton plugin reporting "Unable to bind to INADDR_ANY:33061 (errno=98) - I know, its binding it itself.

2021-09-02T20:01:27.073120Z 2 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2021-09-02T20:01:27.074764Z 2 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv4 localhost address to the whitelist. It is mandatory that it is added.'
2021-09-02T20:01:27.074782Z 2 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv6 localhost address to the whitelist. It is mandatory that it is added.'
2021-09-02T20:01:27.076419Z 11 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. $
2021-09-02T20:01:57.080652Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Timeout while waiting for the group communication engine's communications status to change!'
2021-09-02T20:01:57.080713Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error joining the group while waiting for the network layer to become ready.'
2021-09-02T20:02:07.260221Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'
2021-09-02T20:02:12.341620Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Unable to bind to INADDR_ANY:33061 (socket=346, errno=98)!'
2021-09-02T20:02:12.341674Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Unable to announce tcp port 33061. Port already in use?'
2021-09-02T20:02:12.341709Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error joining the group while waiting for the network layer to become ready.'
2021-09-02T20:02:12.398337Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'
2021-09-02T20:02:17.479137Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Unable to bind to INADDR_ANY:33061 (socket=360, errno=98)!'
2021-09-02T20:02:17.479204Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Unable to announce tcp port 33061. Port already in use?'



Important settings from the my.cnf file

gtid_mode = ON
enforce_gtid_consistency = ON
master_info_repository = TABLE
relay_log_info_repository = TABLE
binlog_checksum = NONE
log_slave_updates = ON
log_bin = binlog
binlog_format = ROW
transaction_write_set_extraction = XXHASH64
loose-group_replication_bootstrap_group = OFF
loose-group_replication_start_on_boot = OFF
loose-group_replication_ssl_mode = REQUIRED
loose-group_replication_recovery_use_ssl = 1

loose-group_replication_single_primary_mode = OFF
loose-group_replication_enforce_update_everywhere_checks = ON

server_id = 1
bind-address = "10.10.0.100"
report_host = "10.10.0.100"
loose-group_replication_local_address = "10.10.0.100:33061"

# Shared replication group configuration
loose-group_replication_group_name = "........"
loose-group_replication_ip_whitelist = "10.10.0.100,10.10.0.101"
loose-group_replication_group_seeds = "10.10.0.100:33061,10.10.0.101:33061"


Would be greatful if anyone could shed any light on getting MySQL GR running again...