MySQL Forums
Forum List  »  InnoDB clusters

Mysql Router Error Too many open files
Posted by: Yevhen Pavliuchek
Date: March 10, 2024 08:24AM

run into issue - Too many open files
centos, docker swarm, innodb cluster 8.0.36 , mysql router 8.0.36 as one service with two task-containers

default settings of innodb cluster with limits innodb_open_files 4000 and open_files_limit 10000

os
ulimit -n
1024

one of container dont have errors in log, the other one have and finish its work after them

is this issue connected to limit of OS or what ?

why '/tmp/mysqlrouter/data/state.json' for writing: Too many open files
this is local file of mysql router container isnt it ? - why it has so many queries ?

or problem with mysql instance
failed to open socket for 0.0.0.0:6446: Too many open files
isn't this problem already not with container router but with mysql db ?
reached limit 1024 on both servers ? couse this containers run on different nodes of cluster swarm

swarm stack:
mysql-router:
hostname: "mysql-router{{.Task.Slot}}"
entrypoint:
- /bin/sh
- -c
- |
exec /run.sh mysqlrouter
environment:
MYSQL_HOST: mysql1
MYSQL_PORT: 3306
MYSQL_USER: inno
MYSQL_PASSWORD: ${DB_ROOT_PASSWORD}
MYSQL_INNODB_CLUSTER_MEMBERS: 2
# MYSQL_CREATE_ROUTER_USER: 0
MYSQL_ROUTER_BOOTSTRAP_EXTRA_OPTIONS: --conf-set-option=DEFAULT.max_connections=2048 --conf-set-option=DEFAULT.max_total_connections=2048
healthcheck:
<<: *common-healthcheck
test: mysqladmin ping -h127.0.0.1 -P 6446 -u$$MYSQL_USER -p$$MYSQL_PASSWORD

log container
loop until stop container

2024-03-09 03:12:23 metadata_cache INFO [7fc2c0213700] Potential changes detected in cluster after metadata refresh (view_id=0)
2024-03-09 03:12:23 metadata_cache INFO [7fc2c0213700] Metadata for cluster 'Cluster' has 3 member(s), single-primary:
2024-03-09 03:12:23 metadata_cache INFO [7fc2c0213700] mysql1:3306 / 33060 - mode=RW
2024-03-09 03:12:23 metadata_cache INFO [7fc2c0213700] mysql2:3306 / 33060 - mode=RO
2024-03-09 03:12:23 metadata_cache INFO [7fc2c0213700] mysql3:3306 / 33060 - mode=RO
2024-03-09 03:12:23 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_ro listening on '0.0.0.0:6449'
2024-03-09 03:12:23 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_rw listening on '0.0.0.0:6448'
2024-03-09 03:12:23 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_ro listening on '0.0.0.0:6447'
2024-03-09 03:12:23 routing INFO [7fc2c0213700] Disconnecting client 127.0.0.1:59852 from server
2024-03-09 03:12:23 routing INFO [7fc2c0213700] Disconnecting client 10.0.1.155:49014 from server
2024-03-09 03:12:23 routing INFO [7fc2c0213700] Routing routing:bootstrap_rw listening on '0.0.0.0:6446' got request to disconnect 2 invalid connections: metadata change
2024-03-09 03:12:23 routing WARNING [7fc2c0213700] [routing:bootstrap_rw] failed to open socket for 0.0.0.0:6446: Too many open files
2024-03-09 03:12:23 metadata_cache INFO [7fc2c0213700] Failed refreshing metadata: Could not open dynamic state file '/tmp/mysqlrouter/data/state.json' for writing: Too many open files
2024-03-09 03:12:23 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_ro listening on 0.0.0.0:6449
2024-03-09 03:12:23 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_rw listening on 0.0.0.0:6448
2024-03-09 03:12:23 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_ro listening on 0.0.0.0:6447
2024-03-09 03:12:23 routing ERROR [7fc2c1257700] connecting to backend failed: not set (destinations:1)
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] Potential changes detected in cluster after metadata refresh (view_id=0)
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] Metadata for cluster 'Cluster' has 3 member(s), single-primary:
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] mysql1:3306 / 33060 - mode=RW
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] mysql2:3306 / 33060 - mode=RO
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] mysql3:3306 / 33060 - mode=RO
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_ro listening on '0.0.0.0:6449'
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_rw listening on '0.0.0.0:6448'
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_ro listening on '0.0.0.0:6447'
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Disconnecting client 10.0.1.155:49014 from server
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Routing routing:bootstrap_rw listening on '0.0.0.0:6446' got request to disconnect 1 invalid connections: metadata change
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_rw listening on '0.0.0.0:6446'
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] Failed refreshing metadata: Could not open dynamic state file '/tmp/mysqlrouter/data/state.json' for writing: Too many open files
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_ro listening on 0.0.0.0:6449
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_rw listening on 0.0.0.0:6448
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_ro listening on 0.0.0.0:6447
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_rw listening on 0.0.0.0:6446
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] Potential changes detected in cluster after metadata refresh (view_id=0)
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] Metadata for cluster 'Cluster' has 3 member(s), single-primary:
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] mysql1:3306 / 33060 - mode=RW
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] mysql2:3306 / 33060 - mode=RO
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] mysql3:3306 / 33060 - mode=RO
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_ro listening on '0.0.0.0:6449'
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_rw listening on '0.0.0.0:6448'
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_ro listening on '0.0.0.0:6447'
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Disconnecting client 10.0.1.155:49014 from server
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Routing routing:bootstrap_rw listening on '0.0.0.0:6446' got request to disconnect 1 invalid connections: metadata change
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_rw listening on '0.0.0.0:6446'
2024-03-09 03:12:24 metadata_cache INFO [7fc2c0213700] Failed refreshing metadata: Could not open dynamic state file '/tmp/mysqlrouter/data/state.json' for writing: Too many open files
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_ro listening on 0.0.0.0:6449
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_rw listening on 0.0.0.0:6448
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_ro listening on 0.0.0.0:6447
2024-03-09 03:12:24 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_rw listening on 0.0.0.0:6446
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] Potential changes detected in cluster after metadata refresh (view_id=0)
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] Metadata for cluster 'Cluster' has 3 member(s), single-primary:
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] mysql1:3306 / 33060 - mode=RW
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] mysql2:3306 / 33060 - mode=RO
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] mysql3:3306 / 33060 - mode=RO
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_ro listening on '0.0.0.0:6449'
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_rw listening on '0.0.0.0:6448'
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_ro listening on '0.0.0.0:6447'
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Disconnecting client 10.0.1.155:49014 from server
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Routing routing:bootstrap_rw listening on '0.0.0.0:6446' got request to disconnect 1 invalid connections: metadata change
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_rw listening on '0.0.0.0:6446'
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] Failed refreshing metadata: Could not open dynamic state file '/tmp/mysqlrouter/data/state.json' for writing: Too many open files
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_ro listening on 0.0.0.0:6449
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_rw listening on 0.0.0.0:6448
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_ro listening on 0.0.0.0:6447
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_rw listening on 0.0.0.0:6446
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] Potential changes detected in cluster after metadata refresh (view_id=0)
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] Metadata for cluster 'Cluster' has 3 member(s), single-primary:
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] mysql1:3306 / 33060 - mode=RW
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] mysql2:3306 / 33060 - mode=RO
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] mysql3:3306 / 33060 - mode=RO
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_ro listening on '0.0.0.0:6449'
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_x_rw listening on '0.0.0.0:6448'
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_ro listening on '0.0.0.0:6447'
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Disconnecting client 10.0.1.155:49014 from server
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Routing routing:bootstrap_rw listening on '0.0.0.0:6446' got request to disconnect 1 invalid connections: metadata change
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Start accepting connections for routing routing:bootstrap_rw listening on '0.0.0.0:6446'
2024-03-09 03:12:25 metadata_cache INFO [7fc2c0213700] Failed refreshing metadata: Could not open dynamic state file '/tmp/mysqlrouter/data/state.json' for writing: Too many open files
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_ro listening on 0.0.0.0:6449
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_x_rw listening on 0.0.0.0:6448
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_ro listening on 0.0.0.0:6447
2024-03-09 03:12:25 routing INFO [7fc2c0213700] Stop accepting connections for routing routing:bootstrap_rw listening on 0.0.0.0:6446
2024-03-09 03:12:25 main SYSTEM [7fc2c2820700] Stopping 'MySQL Router', version: 8.0.36 (MySQL Community - GPL), reason: REQUESTED (Signal Terminated sent by UID: 0 and PID: 0)
2024-03-09 03:12:26 routing INFO [7fc28a7fc700] [routing:bootstrap_x_rw] stopped
2024-03-09 03:12:26 routing INFO [7fc28affd700] [routing:bootstrap_x_ro] stopped
2024-03-09 03:12:26 routing INFO [7fc28bfff700] [routing:bootstrap_ro] stopped
2024-03-09 03:12:26 routing ERROR [7fc2c0a56700] connecting to backend failed: not set (destinations:1)
2024-03-09 03:12:26 routing INFO [7fc28b7fe700] [routing:bootstrap_rw] stopped



Edited 1 time(s). Last edit at 03/10/2024 08:34AM by Yevhen Pavliuchek.

Options: ReplyQuote


Subject
Views
Written By
Posted
Mysql Router Error Too many open files
765
March 10, 2024 08:24AM


Sorry, only registered users may post in this forum.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.