MySQL Forums
Forum List  »  NDB clusters

Re: MySQL Cluster data nodes fails with 7200: LCP fragment scan watchdog
Posted by: Puneet N
Date: March 08, 2018 08:27PM

Hi Mikael,
It was indeed the issue. We had a RBD from CEPH cluster assigned as the drive for mysql and during the deep scrub cycles of CEPH the I/O stalled for mysql to write its data.
After moving out the CEPH completely from the picture the cluster worked flawleslly until few days ago when one of the nodes crashed again.

The error is as follows:
---------------------------------------------------------------------------------
Time: Monday 5 March 2018 - 16:00:18
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19385) 0x00000002
Program: ndbd
Pid: 8762
Version: mysql-5.6.28 ndb-7.4.10
Trace: /var/lib/mysql/mycluster_data/ndb_3_trace.log.2 [t1..t1]
***EOM***
---------------------------------------------------------------------------------

While i was trying to bring this node up again with "ndbd" i got the above error on the first node and now both nodes are down.
The error that i got this time for the second node was as follows:
---------------------------------------------------------------------------------
Time: Thursday 8 March 2018 - 17:38:24
Status: Temporary error, restart node
Message: Another node failed during system restart, please investigate error(s) on other node(s) (Restart error)
Error: 2308
Error data: Node 2 disconnected
Error object: QMGR (Line: 3473) 0x00000002
Program: ndbd
Pid: 24411
Version: mysql-5.6.28 ndb-7.4.10
Trace: /var/lib/mysql/mycluster_data/ndb_3_trace.log.3 [t1..t1]
***EOM***
---------------------------------------------------------------------------------

Trying to find what could be the issue for the first error message to come and not getting any leads over the Internet.
Can you please help me understand what could be the issue here?
Please let me know if you need more details.

Options: ReplyQuote




Sorry, only registered users may post in this forum.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.