Failure to restore schema (Resource configuration error) Error: 2355

[Parimal Patel]

Hi,

I have set up NDB Cluster with 4 nodes.

NodeA : SQLNode1, DataNode1

NodeB : SQLNode2, DataNode2

NodeC : Mgmt Node1

NodeD : Mgmt Node2

I tried to shutdown all VMs on which these nodes were running and restarted them with the order specifed as below:

1) NodeC
2) NodeD
3) NodeA
4) NodeB

The management process and mysqld were started successfully. But ndbmtd processes were crashing suddenly after starting.

I have pasted logs from NodeA's ndbmtd's ndb_1_error.log file as below:
------------------------------------------------------------------------
Time: Wednesday 7 May 2014 - 12:47:42
St1497: Permanent error, external action needed
Message: Failure to restore schema (Resource configuration error)
Error: 2355
Error data: Failed to recreate object 510 during restart, error 708.
Error object: DBDICT (Line: 4688) 0x00000000
Program: ndbmtd
Pid: 2163 thr: 0
Version: mysql-5.6.17 ndb-7.3.5
Trace: /home/mysql/mysqlcluster_data/1//ndb_1_trace.log.10 [t1..t4]
***EOM***

Time: Wednesday 7 May 2014 - 12:47:49
Status: Permanent error, external action needed
Message: Failure to restore schema (Resource configuration error)
Error: 2355
Error data: Failed to recreate object 510 during restart, error 708.
Error object: DBDICT (Line: 4688) 0x00000000
Program: ndbmtd
Pid: 2200 thr: 0
Version: mysql-5.6.17 ndb-7.3.5
Trace: /home/mysql/mysqlcluster_data/1//ndb_1_trace.log.11 [t1..t4]
***EOM***

Time: Wednesday 7 May 2014 - 12:47:57
Status: Permanent error, external action needed
Message: Failure to restore schema (Resource configuration error)
Error: 2355
Error data: Failed to recreate object 510 during restart, error 708.
Error object: DBDICT (Line: 4688) 0x00000000
Program: ndbmtd
Pid: 2238 thr: 0
Version: mysql-5.6.17 ndb-7.3.5
Trace: /home/mysql/mysqlcluster_data/1//ndb_1_trace.log.12 [t1..t4]
***EOM***

------------------------------------------------------------------------

Can you please help me on this ?

One more thing is that when I tried --initial option when starting ndbmtd
/usr/local/mysql/bin/ndbmtd --ndb-nodeid=1 --ndb-connectstring=nodec:1186,noded:1186,


The data was wiped out from all the dbs but the tablespace and undo files were still on place but not been shown in Information_schema.FILES .. ?

Can I recover this data somehow from filesystem ?