2355: 'Failure to restore schema(Resource configuration error)
After rebooting data nodes we are getting this error
error: [ code: 1509 line: 22171 node: 2 count: 1 status: 0 key: 0 name: '' ]
2015-04-07 12:06:10 [ndbd] INFO -- Failed to restore schema during restart, error 1509.
2015-04-07 12:06:10 [ndbd] INFO -- DBDICT (Line: 4303) 0x00000000
2015-04-07 12:06:10 [ndbd] INFO -- Error handler restarting system
2015-04-07 12:06:10 [ndbd] INFO -- Error handler shutdown completed - exiting
2015-04-07 12:06:10 [ndbd] ALERT -- Angel detected too many startup failures(3), not restarting again
2015-04-07 12:06:10 [ndbd] ALERT -- Node 2: Forced node shutdown completed. Occured during startphase 4. Caused by error 2355: 'Failure to restore schema(Resource configuration error). Permanent error, external action needed'.
$cat ndb_2_error.log
Time: Tuesday 7 April 2015 - 12:06:10
Status: Permanent error, external action needed
Message: Failure to restore schema (Resource configuration error)
Error: 2355
Error data: Failed to restore schema during restart, error 1509.
Error object: DBDICT (Line: 4303) 0x00000000
Program: ndbmtd
Pid: 3530 thr: 0
Version: mysql-5.6.17 ndb-7.3.5
Trace: /home/mysql/mysqlcluster_data/2//ndb_2_trace.log.3 [t1..t5]
***EOM***
Trace Files
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 2, r.sigId: 2078 gsn: 264 "FSREADREQ" prio: 0
s.bn: 247/1 "DBLQH", s.proc: 2, s.sigId: 4000951 length: 8 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
FilePointer: 13
UserReference: H'02f70002 Operation flag: H'00000000 (No sync, Format=List of pairs)
varIndex: 1
numberOfPages: 1
pageData: H'00000000, H'00001380
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 2, r.sigId: 2077 gsn: 264 "FSREADREQ" prio: 0
s.bn: 247/1 "DBLQH", s.proc: 2, s.sigId: 4000950 length: 8 trace: 0 #sec: 0 fragInf: 0
UserPointer: 1
FilePointer: 12
UserReference: H'02f70002 Operation flag: H'00000000 (No sync, Format=List of pairs)
varIndex: 1
numberOfPages: 1
pageData: H'00000001, H'00001380
Configuration File in Management Node
[NDB_MGMD DEFAULT]
Portnumber=1186
[NDB_MGMD]
NodeId=49
HostName=hazelcasta
DataDir=/home/mysql/mysqlcluster_data/49/
Portnumber=1186
[NDB_MGMD]
NodeId=52
HostName=hazelcastb
DataDir=/home/mysql/mysqlcluster_data/52/
Portnumber=1186
[TCP DEFAULT]
SendBufferMemory=4M
ReceiveBufferMemory=4M
[NDBD DEFAULT]
BackupMaxWriteSize=1M
BackupDataBufferSize=16M
BackupLogBufferSize=4M
BackupMemory=20M
BackupReportFrequency=10
MemReportFrequency=30
LogLevelStartup=15
LogLevelShutdown=15
LogLevelCheckpoint=8
LogLevelNodeRestart=15
DataMemory=2000M
IndexMemory=512M
MaxNoOfTables=4096
MaxNoOfTriggers=10000
NoOfReplicas=2
StringMemory=5M
MaxNoOfSubscribers=20000
DiskPageBufferMemory=512M
SharedGlobalMemory=768M
LongMessageBuffer=32M
MaxNoOfConcurrentOperations=1100000
MaxNoOfConcurrentTransactions=900000
MaxNoOfLocalOperations=1000000
MaxNoOfConcurrentScans=400
MaxNoOfLocalScans=800
BatchSizePerLocalScan=512
FragmentLogFileSize=256M
NoOfFragmentLogFiles=25
RedoBuffer=32M
DiskIOThreadPool=16
StopOnError=false
TransactionInactiveTimeout=180000
MaxNoOfOrderedIndexes=1000
MaxNoOfUniqueHashIndexes=2000
MaxNoOfAttributes=8000
TransactionDeadlockDetectionTimeout=1800000
LcpScanProgressTimeout=600
TimeBetweenWatchdogCheck=300000
TimeBetweenWatchdogCheckInitial=300000
LockPagesInMainMemory=2
MaxBufferedEpochs=5000
RealtimeScheduler=1
ThreadConfig=ldm={cpubind=2},rep={cpubind=3},tc={cpubind=4},recv={cpubind=5},send={cpubind=6},main={cpubind=7},io={cpubind=7}
NoOfFragmentLogParts=4
MinFreePct=5
TwoPassInitialNodeRestartCopy=true
BuildIndexThreads=8
RedoOverCommitCounter=5
RedoOverCommitLimit=60
TimeBetweenEpochsTimeout=0
Please help.....