MySQL 8 Group Replication very slow with inserts

[Icaro Orte]

Hi there

I have a Group Replication with MySQL Server version 8.0.19, master master with 2 servers (FreeBSD 12.1). The system works great, but there is a problem when inserting/updating, making queries very slow

For example, I have a script that perfoms 500 inserts. With group replication, it takes 50 seconds to finish. If I stop group replication in the other server, it takes only 1 second.

Servers are in different datacenters, but ping is about 50ms.

My cnf is as follows (similar in both servers):

[mysqld]
server-id = 10
auto_increment_increment = 2
auto_increment_offset = 1
bind-address = 0.0.0.0
report-host = xx.xx.xx.xx
gtid_mode = ON
enforce_gtid_consistency = ON
default_storage_engine = InnoDB
basedir = /usr/local
user = mysql
port = 3307
socket = /tmp/mysqll.sock
mysqlx_socket = /tmp/mysqllx.sock
pid-file = /chroot/mysqll/mysqll.pid
datadir = /chroot/mysqll/data
tmpdir = /chroot/mysqll/data_tmpdir
secure-file-priv = /chroot/mysqll/data_secure
innodb_buffer_pool_size = 2000M
log_error = /logs/mysqll-error.log
slow_query_log_file = /logs/mysqll-slow.log
log-bin = /logs/mysqll/mysql-bin
log-output = TABLE
master-info-repository = TABLE
relay-log-info-repository = TABLE
relay-log-recovery = 1
sync_binlog = 1
sync_relay_log = 1
binlog_cache_size = 16M
binlog_checksum = NONE
binlog_format = ROW
log_slave_updates = ON
expire_logs_days = 3
default_password_lifetime = 0
gtid-mode = ON
enforce_gtid_consistency = ON
safe-user-create = 1
lower_case_table_names = 1
explicit-defaults-for-timestamp = 1
open_files_limit = 32768
table_open_cache = 16384
table_definition_cache = 8192
net_retry_count = 16384
key_buffer_size = 256M
max_allowed_packet = 64M
long_query_time = 2
slow_query_log = 1
skip-symbolic-links
innodb_flush_method = O_DIRECT
innodb_log_file_size = 256M
innodb_log_buffer_size = 16M
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_autoinc_lock_mode = 2

default_authentication_plugin = mysql_native_password
sql_mode = NO_ENGINE_SUBSTITUTION

max_connections = 1000
max_user_connections = 0

# I tried the following settings, but didn't get an improvement either
#slave_parallel_type = LOGICAL_CLOCK
#slave_parallel_workers = 40
#slave_preserve_commit_order = ON
#sync_binlog=0
#innodb_flush_log_at_trx_commit=2
#innodb_flush_log_at_timeout=60
#group_replication_compression_threshold=100
#group_replication_poll_spin_loops=1000
#group_replication_gtid_assignment_block_size=1000000
#binlog_row_image=MINIMAL
#group_replication_enforce_update_everywhere_checks=OFF

plugin-load = group_replication.so
plugin-load-add = mysql_clone.so
transaction_write_set_extraction = XXHASH64
loose-group_replication_group_name = "e66cedbb-3ea7-11ea-aad0-000c293a6000"
loose-group_replication_start_on_boot = on
loose-group_replication_local_address = "xx.xx.xx.xx:33062"
loose-group_replication_group_seeds = "xx.xx.xx.xx:33062,xx.xx.xx.xx:33062"
loose-group_replication_bootstrap_group = off
loose-group_replication_single_primary_mode = FALSE
loose-group_replication_enforce_update_everywhere_checks = TRUE
loose-group_replication_ip_whitelist = "xx.xx.xx.xx,xx.xx.xx.xx"
loose-group_replication_recovery_retry_count = 3
loose-group_replication_recovery_reconnect_interval = 120
loose-group_replication_ssl_mode = REQUIRED
loose-group_replication_recovery_use_ssl = 1


[mysqldump]
max_allowed_packet = 256M
quote_names
quick


----------------------

How could I optimize it so that it doesn't have that big drop in performance? Could the connection be more asynchronous so that you don't have to wait for the other server?

I appreciate your comments

Best regards,
Icaro