'Cluster Failure' from NDB. Could not acquire global schema lock
Hi all,
I'm new to MySQL Cluster and started evaluating MySQL Cluster Solution.
I'm using 2 datanodes, 1 SQL Node and 1 API nodes for MySQL Cluster setup.
Data Node and SQL configuration:
================================
[mysqld]
ndbcluster
# IP address of the cluster management node
ndb-connectstring=1.0.0.114
[mysql_cluster]
# IP address of the cluster management node
ndb-connectstring=1.0.0.114
Mgmt Node configuration:
========================
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
#DataMemory=80M # How much memory to allocate for data storage
#IndexMemory=18M # How much memory to allocate for index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the "world" database takes up
# only about 500KB, this should be more than enough for
# this example Cluster setup.
[tcp default]
# TCP/IP options:
#portnumber=2202 # This the default; however, you can use any
# port that is free for all the hosts in the cluster
# Note: It is recommended that you do not specify the port
# number at all and simply allow the default value to be used
# instead
[ndb_mgmd]
# Management process options:
hostname=1.0.0.114 # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster # Directory for MGM node log files
[ndbd]
NodeId:10
# Options for data node "A":
# (one [ndbd] section per data node)
hostname=1.0.0.111 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
[ndbd]
NodeId:11
# Options for data node "B":
hostname=1.0.0.112 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
[mysqld]
NodeId:20
# SQL node options:
hostname=1.0.0.113 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
mgmt node output:
=================
[root@localhost mysql]# ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=10 @1.0.0.111 (mysql-5.6.17 ndb-7.3.5, Nodegroup: 0, *)
id=11 @1.0.0.112 (mysql-5.6.17 ndb-7.3.5, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @1.0.0.114 (mysql-5.6.17 ndb-7.3.5)
[mysqld(API)] 1 node(s)
id=20 @1.0.0.113 (mysql-5.6.17 ndb-7.3.5)
[root@localhost mysql]#
the show processlist output in data and SQL nodes:
Here we could see the state is still waiting.
mysql> show processlist;
+----+-------------+-----------+------+---------+------+-----------------------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-------------+-----------+------+---------+------+-----------------------------------+------------------+
| 1 | system user | | | Daemon | 0 | Waiting for event from ndbcluster | NULL |
| 2 | root | localhost | ss | Query | 0 | init | show processlist |
+----+-------------+-----------+------+---------+------+-----------------------------------+------------------+
2 rows in set(0.00 sec).
On creating the database and a table via the SQL Node. Could see the below message getting displayed in the data nodes.
mysql> show warnings;
+---------+------+---------------------------------------------------------------------------------+
| Level | Code | Message |
+---------+------+---------------------------------------------------------------------------------+
| Warning | 1296 | Got error 4009 'Cluster Failure' from NDB. Could not acquire global schema lock |
+---------+------+---------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql>
Datanode output for global status:
mysql> show global status like 'ndb_number_of%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| Ndb_number_of_data_nodes | 2 |
| Ndb_number_of_ready_data_nodes | 0 |
+--------------------------------+-------+
2 rows in set (0.00 sec)
mysql>
SQL node output for global status:
mysql> show global status like 'ndb_number_of%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| Ndb_number_of_data_nodes | 2 |
| Ndb_number_of_ready_data_nodes | 2 |
+--------------------------------+-------+
2 rows in set (0.00 sec)
I can't able view the SQL changes(create databases/tables) done from SQL Node in the data nodes.
MySQL Cluster logs:
===================
2014-07-02 14:25:19 [MgmtSrvr] INFO -- Got initial configuration from '/var/lib/mysql-cluster/config.ini', will try to set it when all ndb_mgmd(s) started
2014-07-02 14:25:19 [MgmtSrvr] INFO -- Id: 1, Command port: *:1186
2014-07-02 14:25:19 [MgmtSrvr] INFO -- Node 1: Node 1 Connected
2014-07-02 14:25:19 [MgmtSrvr] INFO -- MySQL Cluster Management Server mysql-5.6.17 ndb-7.3.5 started
2014-07-02 14:25:19 [MgmtSrvr] INFO -- Node 1 connected
2014-07-02 14:25:19 [MgmtSrvr] INFO -- Starting initial configuration change
2014-07-02 14:25:19 [MgmtSrvr] INFO -- Configuration 1 commited
2014-07-02 14:25:19 [MgmtSrvr] INFO -- Config change completed! New generation: 1
2014-07-02 14:25:45 [MgmtSrvr] INFO -- Nodeid 10 allocated for NDB at 1.0.0.111
2014-07-02 14:25:45 [MgmtSrvr] INFO -- Node 1: Node 10 Connected
2014-07-02 14:25:45 [MgmtSrvr] INFO -- Node 10: Buffering maximum epochs 100
2014-07-02 14:25:45 [MgmtSrvr] INFO -- Node 10: Start phase 0 completed
2014-07-02 14:25:45 [MgmtSrvr] INFO -- Node 10: Communication to Node 11 opened
2014-07-02 14:25:45 [MgmtSrvr] INFO -- Node 10: Waiting 30 sec for nodes 11 to connect, nodes [ all: 10 and 11 connected: 10 no-wait: ]
2014-07-02 14:25:48 [MgmtSrvr] INFO -- Node 10: Waiting 27 sec for nodes 11 to connect, nodes [ all: 10 and 11 connected: 10 no-wait: ]
2014-07-02 14:25:51 [MgmtSrvr] INFO -- Node 10: Waiting 24 sec for nodes 11 to connect, nodes [ all: 10 and 11 connected: 10 no-wait: ]
2014-07-02 14:25:53 [MgmtSrvr] INFO -- Nodeid 11 allocated for NDB at 1.0.0.112
2014-07-02 14:25:53 [MgmtSrvr] INFO -- Node 1: Node 11 Connected
2014-07-02 14:25:53 [MgmtSrvr] INFO -- Node 11: Buffering maximum epochs 100
2014-07-02 14:25:53 [MgmtSrvr] INFO -- Node 11: Start phase 0 completed
2014-07-02 14:25:53 [MgmtSrvr] INFO -- Node 11: Communication to Node 10 opened
2014-07-02 14:25:53 [MgmtSrvr] INFO -- Node 11: Waiting 30 sec for nodes 10 to connect, nodes [ all: 10 and 11 connected: 11 no-wait: ]
2014-07-02 14:25:54 [MgmtSrvr] INFO -- Node 10: Node 11 Connected
2014-07-02 14:25:54 [MgmtSrvr] INFO -- Node 11: Node 10 Connected
2014-07-02 14:25:54 [MgmtSrvr] INFO -- Node 10: Start with all nodes 10 and 11
2014-07-02 14:25:54 [MgmtSrvr] INFO -- Node 10: CM_REGCONF president = 10, own Node = 10, our dynamic id = 0/1
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 11: CM_REGCONF president = 10, own Node = 11, our dynamic id = 0/2
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 10: Node 11: API mysql-5.6.17 ndb-7.3.5
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 11: Node 10: API mysql-5.6.17 ndb-7.3.5
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 11: Start phase 1 completed
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 10: Start phase 1 completed
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 11: Start phase 2 completed (system restart)
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 10: Start phase 2 completed (system restart)
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 10: Start phase 3 completed (system restart)
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 11: Start phase 3 completed (system restart)
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 10: Restarting cluster to GCI: 3899
2014-07-02 14:25:57 [MgmtSrvr] INFO -- Node 10: Starting to restore schema
2014-07-02 14:25:58 [MgmtSrvr] INFO -- Node 10: Restore of schema complete
2014-07-02 14:25:58 [MgmtSrvr] INFO -- Node 11: Starting to restore schema
2014-07-02 14:25:58 [MgmtSrvr] INFO -- Node 11: Restore of schema complete
2014-07-02 14:25:58 [MgmtSrvr] INFO -- Node 10: DICT: activate index 6 done (sys/def/5/ndb_index_stat_sample_x1)
2014-07-02 14:25:58 [MgmtSrvr] INFO -- Node 10: Node: 10 StartLog: [GCI Keep: 1031 LastCompleted: 3899 NewestRestorable: 3899]
2014-07-02 14:25:58 [MgmtSrvr] INFO -- Node 10: Node: 11 StartLog: [GCI Keep: 1031 LastCompleted: 3899 NewestRestorable: 3899]
2014-07-02 14:26:00 [MgmtSrvr] INFO -- Node 11: LQH: Starting to rebuild ordered indexes
2014-07-02 14:26:00 [MgmtSrvr] INFO -- Node 11: LQH: index 6 rebuild done
2014-07-02 14:26:00 [MgmtSrvr] INFO -- Node 11: LQH: Rebuild ordered indexes complete
2014-07-02 14:26:00 [MgmtSrvr] INFO -- Node 10: LQH: Starting to rebuild ordered indexes
2014-07-02 14:26:00 [MgmtSrvr] INFO -- Node 10: LQH: index 6 rebuild done
2014-07-02 14:26:00 [MgmtSrvr] INFO -- Node 10: LQH: Rebuild ordered indexes complete
2014-07-02 14:26:00 [MgmtSrvr] INFO -- Node 10: Start phase 4 completed (system restart)
2014-07-02 14:26:00 [MgmtSrvr] INFO -- Node 11: Start phase 4 completed (system restart)
2014-07-02 14:26:00 [MgmtSrvr] INFO -- Node 10: GCP Monitor: unlimited lags allowed
2014-07-02 14:26:01 [MgmtSrvr] INFO -- Node 10: Local checkpoint 4 started. Keep GCI = 2650 oldest restorable GCI = 1370
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Local checkpoint 4 completed
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Start phase 5 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Start phase 5 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Start phase 6 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Start phase 6 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: President restarts arbitration thread [state=1]
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Start phase 7 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Start phase 7 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Start phase 8 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Start phase 8 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Start phase 9 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Start phase 9 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Start phase 100 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Start phase 100 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Start phase 101 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Start phase 101 completed (system restart)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Started (mysql-5.6.17 ndb-7.3.5)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Started (mysql-5.6.17 ndb-7.3.5)
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Node 1: API mysql-5.6.17 ndb-7.3.5
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Node 1: API mysql-5.6.17 ndb-7.3.5
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Prepare arbitrator node 1 [ticket=733f0001426bb9c9]
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Started arbitrator node 1 [ticket=733f0001426bb9c9]
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Communication to Node 20 opened
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Communication to Node 20 opened
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Node 20 Connected
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Node 20 Connected
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 10: Node 20: API mysql-5.6.17 ndb-7.3.5
2014-07-02 14:26:05 [MgmtSrvr] INFO -- Node 11: Node 20: API mysql-5.6.17 ndb-7.3.5
Please let me know the issue/errors here. Please do the needful.