mysql cluster failure - Error while reading the REDO log

[Alex Proca]

Hi all,

I'm having a headache giving problem with a mysql cluster running 5.0.19 on Debian 7 x64.

Current:
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 (not connected, accepting connect from 86.107.69.174)
id=3 (not connected, accepting connect from 95.162.65.36)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @93.114.175.230 (Version: 5.0.19)

[mysqld(API)] 2 node(s)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)


After finding the cluster in the below situation:
- NDBD servers disconnected from the management server
- APIs connected to management server
- NDBD node 3 had latest data / NDBD node 2 had older data

Usually when this happens and a cluster reset fixes the problem. After I tried to start the NDBD I get this on node 3:

from management server log :

2015-07-03 12:24:29 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 reserved for ip 95.162.65.36, m_reserved_nodes 000000000000000a.
2015-07-03 12:24:30 [MgmSrvr] INFO -- Node 1: Node 3 Connected
2015-07-03 12:24:30 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 0000000000000002.
2015-07-03 12:25:03 [MgmSrvr] INFO -- Node 3: Start phase 1 completed
2015-07-03 12:26:03 [MgmSrvr] INFO -- Node 3: Start phase 2 completed (system restart)
2015-07-03 12:26:03 [MgmSrvr] INFO -- Node 3: Start phase 3 completed (system restart)
-------
2015-07-03 12:27:14 [MgmSrvr] WARNING -- Node 3: DICT: index 243 activated failed: code=4242 line=0 node=753
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 244 activated done
2015-07-03 12:27:14 [MgmSrvr] WARNING -- Node 3: DICT: index 245 activated failed: code=4242 line=0 node=758
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 247 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 249 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 251 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 253 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 255 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 256 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 260 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 263 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 265 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 267 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 269 activated done
2015-07-03 12:27:14 [MgmSrvr] INFO -- Node 3: DICT: index 270 activated done
2015-07-03 12:27:14 [MgmSrvr] WARNING -- Node 3: DICT: index 271 activated failed: code=4242 line=0 node=807
2015-07-03 12:27:46 [MgmSrvr] INFO -- Node 1: Node 3 Connected
2015-07-03 12:27:46 [MgmSrvr] ALERT -- Node 3: Forced node shutdown completed. Occured during startphase 4. Initiated by signal 0. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.


The node 2 that had older data has been initiated so my only copy of the data is in node 3. Is there any possibility to recover the data even partly ?

I've searched through the bug database with not much luck.

Any help is highly appreciated.

Thanks!

Alex