[Warning] NDB: Failed to acquire global schema lock, error: (4009)Cluster Failure
I have 3-node mysql cluster. 1 cluster management node 2 datanode/API nodes.
Upon changing the length of a column, show warnings advised that i should raise the MaxNoOfOrderedIndex in config.ini. After the change i reloaded the cluster management ndb_mgmd followed by restart of 2nd data/api node. Before the restart process of 2nd node completed, node 1 got crashed. Since then im facing the error. I also have tried to reload management by disabling the parameters added after the initial configuration, but all in vain.
Content of configuration files:
Config.ini
[ndb_mgmd]
#Management process options:
hostname=10.50.13.106 #Hostname of the manager
datadir=/storage/mysql/cluster/logs #Directory for the log files
[ndbd default]
NoOfReplicas=2 #Number of replicas
#MaxNoOfAttributes=16384
DataMemory=29492M
IndexMemory=512M
#MaxNoOfConcurrentOperations=100000
#MaxNoOfOrderedIndexes=256
#SchedulerSpinTimer=400
#SchedulerExecutionTimer=100
#RealTimeScheduler=1
#TimeBetweenGlobalCheckpoints=1000
#TimeBetweenEpochs=200
# Lock table in memory to increase speed
#LockPagesInMainMemory=1
# Use O_DIRECT write
#ODirect=1
#FragmentLogFileSize=256M
# 6*DataMemory/4/FragmentLogFileSize
#NoOfFragmentLogFiles=560
#RedoBuffer=32M
[ndbd]
#datanode
hostname=10.50.13.109 #Hostname of the first data node
datadir=/storage/mysql/cluster/datafiles #Remote directory for the data files
[ndbd]
#datanode
hostname=10.50.13.112 #Hostname of the second data node
datadir=/storage/mysql/cluster/datafiles #Remote directory for the data files
[mysqld]
#SQL Node
hostname=10.50.13.109
[mysqld]
#SQL Node
hostname=10.50.13.112
my.cnf node1:
==============
# For advice on how to change settings please see
#
http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
[mysqld]
bind-address=10.50.13.109
ndbcluster
ndb-connectstring=10.50.13.106 #IP address of Management Node
default_storage_engine=ndbcluster #Database Engine
datadir=/storage/mysql/datafiles
socket=/var/lib/mysql/mysql.sock
skip-name-resolve
#ndb-index-stat-enable=0
#max_connections = 2500
####For Networker########
port=3306
####Performance Schema###
#performance-schema-instrument='statement/%=ON'
#performance-schema-consumer-events-statements-current=ON
#performance-schema-consumer-events-statements-history=ON
#performance-schema-consumer-events-statements-history-long=ON
#performance-schema-consumer-statements-digest=ON
######For Bin logging###
###############################################
#########Disabling binlog for diagnostics######
###############################################
#server-id=2
#log-bin=mysql-bin
#binlog_format = MIXED
#expire_logs_days = 7
#max_binlog_size = 100M
#max_allowed_packet = 32M
#LockPagesInMainMemory=1
# Use O_DIRECT write
#ODirect=1
[mysql_cluster]
ndb-connectstring=10.50.13.106 # IP address of Management Node
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
#datadir=/var/lib/mysql
#socket=/var/lib/mysql/mysql.sock
[mysql_safe]
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
my.cnf node2:
=============
# For advice on how to change settings please see
#
http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
[mysqld]
bind-address=10.50.13.112
ndbcluster
ndb-connectstring=10.50.13.106 #IP address of Management Node
default_storage_engine=ndbcluster #Database Engine
datadir=/storage/mysql/datafiles
socket=/var/lib/mysql/mysql.sock
max_connections = 2500
####For Networker########
port=3306
####Performance Schema###
#performance-schema-instrument='statement/%=ON'
#performance-schema-consumer-events-statements-current=ON
#performance-schema-consumer-events-statements-history=ON
#performance-schema-consumer-events-statements-history-long=ON
#performance-schema-consumer-statements-digest=ON
######For Bin logging###
server-id=3
log-bin=mysql-bin
binlog_format = MIXED
expire_logs_days = 7
max_binlog_size = 100M
max_allowed_packet = 32M
[mysql_cluster]
ndb-connectstring=10.50.13.106 # IP address of Management Node
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
#datadir=/var/lib/mysql
#socket=/var/lib/mysql/mysql.sock
[mysql_safe]
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Any Help in this regard is appreciable.