Re: MySQL Cluster data nodes fails with 7200: LCP fragment scan watchdog

[Puneet N]

Hi Mikael,
It was indeed the issue. We had a RBD from CEPH cluster assigned as the drive for mysql and during the deep scrub cycles of CEPH the I/O stalled for mysql to write its data.
After moving out the CEPH completely from the picture the cluster worked flawleslly until few days ago when one of the nodes crashed again.

The error is as follows:
---------------------------------------------------------------------------------
Time: Monday 5 March 2018 - 16:00:18
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19385) 0x00000002
Program: ndbd
Pid: 8762
Version: mysql-5.6.28 ndb-7.4.10
Trace: /var/lib/mysql/mycluster_data/ndb_3_trace.log.2 [t1..t1]
***EOM***
---------------------------------------------------------------------------------

While i was trying to bring this node up again with "ndbd" i got the above error on the first node and now both nodes are down.
The error that i got this time for the second node was as follows:
---------------------------------------------------------------------------------
Time: Thursday 8 March 2018 - 17:38:24
Status: Temporary error, restart node
Message: Another node failed during system restart, please investigate error(s) on other node(s) (Restart error)
Error: 2308
Error data: Node 2 disconnected
Error object: QMGR (Line: 3473) 0x00000002
Program: ndbd
Pid: 24411
Version: mysql-5.6.28 ndb-7.4.10
Trace: /var/lib/mysql/mycluster_data/ndb_3_trace.log.3 [t1..t1]
***EOM***
---------------------------------------------------------------------------------

Trying to find what could be the issue for the first error message to come and not getting any leads over the Internet.
Can you please help me understand what could be the issue here?
Please let me know if you need more details.