Error starting data node with --initial

[Kent Williams]

We have a small cluster as follows:

Node 1: Management node
Node 2: Data + MySQL node
Node 3: Data + MySQL node

Node 2 had a drive failure. After replacing the drive, ndbmtd was started with --initial, however it kept failing during startup stage 5.

At that point in time, the cluster was running version 7.2.7.

We have since upgraded Node 1 (management) and Node 2 (failed node) to 7.6.12 with the view to upgrading Node 3 once Node 2 was back online.

Unfortunately, ndbmtd --initial is still failing to start, and it appears to be at the same point. See below for the error output. I'm not familiar enough with NDB to follow the trace log file.

Any thoughts on how we could get Node 2 started?

Time: Monday 28 October 2019 - 01:22:31
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbdihMain.cpp
Error object: DBDIH (Line: 19277) 0x00000006 Check (noOfFrags * (rf.rwfTabPtr.p->noOfBackups + 1)) <= cnoFreeReplicaRec failed
Program: ndbmtd
Pid: 2701 thr: 0
Version: mysql-5.7.28 ndb-7.6.12
Trace file name: ndb_2_trace.log.5
Trace file path: /usr/local/mysql/data/ndbmtd/ndb_2_trace.log.5 [t1..t4]
***EOM***