Re: Data Node Won't Start: Internal program error (failed ndbrequire)

[Jeff Steele]

I have gathered some more information about this. There appears to have been a problem with the first backup after the upgrade. However, I didn't notice it. I found this in the error log:

Time: Sunday 25 October 2020 - 00:55:10
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming
error or missing error message, please report a bug)
Error: 2341
Error data: Backup.cpp
Error object: BACKUP (Line: 9682) 0x00000002 Check noOfOps == done failed
Program: ndbmtd
Pid: 6155 thr: 2
Version: mysql-8.0.22 ndb-8.0.22
Trace file name: ndb_3_trace.log.4_t2
Trace file path: /usr/local/mysql/data/ndb_3_trace.log.4 [t1..t7]
***EOM***


My current theory is that the backup caused my primary node to crash Sunday morning. However, the cluster continued running with just the second node. Then, this morning's backup crashed it. When I found the cluster had crashed, I started the primary node first. Because it appears not have not been running for a day, a day's worth of data was missing. The second node probably has the missing data, but it won't start.