Re: Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341
Hi Tiago,
Thanks for the quick response!
Configuration before change (1 mgm node, 2 data nodes and 2 SQL nodes):
[ndb_mgmd default]
# Directory for MGM node log files
DataDir=/var/lib/mysql-cluster
[ndb_mgmd]
#Management Node db1
HostName=172.20.10.50
[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=100M # Memory allocate for data storage
#Directory for Data Node
DataDir=/var/lib/mysql-cluster
[mysqld]
#SQL Node db4
HostName=172.20.10.51
[mysqld]
#SQL Node db5
HostName=172.20.10.52
[ndbd]
#Data Node db2
HostName=172.20.10.54
[ndbd]
#Data Node db3
HostName=172.20.10.53
Configuration after change (1 mgm node, 4 data nodes and 2 SQL nodes):
[ndb_mgmd default]
# Directory for MGM node log files
DataDir=/var/lib/mysql-cluster
[ndb_mgmd]
#Management Node db1
HostName=172.20.10.50
[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=100M # Memory allocate for data storage
#Directory for Data Node
DataDir=/var/lib/mysql-cluster
[mysqld]
#SQL Node db4
HostName=172.20.10.51
[mysqld]
#SQL Node db5
HostName=172.20.10.52
[ndbd]
#Data Node db2
HostName=172.20.10.54
[ndbd]
#Data Node db3
HostName=172.20.10.53
[ndbd]
#Data Node db6
HostName=172.20.10.55
[ndbd]
#Data Node db7
HostName=172.20.10.56
I have only added the last two nodes to include 2 more data nodes. I have followed the instructions from this link as detailed there:
https://dev.mysql.com/doc/refman/8.0/en/mysql-cluster-online-add-node-example.html
The version I am using is 8.0.23. It works with 1 management node, 2 data nodes and 2 SQL nodes!
1. I change the config.ini on mgm node to include two more IP addresses for data nodes.
2. On ndb_mgm> 1 STOP
3. Then I start it with sudo ndb_mgmd -f /var/lib/mysql-cluster/config.ini --reload
4. I goto ndb_mgm and SHOW - it shows:
[ndbd(NDB)] 4 node(s)
id=4 @172.20.10.54 (mysql-8.0.23 ndb-8.0.23, Nodegroup: 0, *)
id=5 @172.20.10.53 (mysql-8.0.23 ndb-8.0.23, Nodegroup: 0)
id=6 (not connected, accepting connect from 172.20.10.55)
id=7 (not connected, accepting connect from 172.20.10.56)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.20.10.50 (mysql-8.0.23 ndb-8.0.23)
[mysqld(API)] 2 node(s)
id=2 @172.20.10.51 (mysql-8.0.23 ndb-8.0.23)
id=3 @172.20.10.52 (mysql-8.0.23 ndb-8.0.23)
I do rolling restart by:
5. 4 RESTART - wait for it to restart
6. 5 RESTART - wait for it to restart
7. On SQL nodes 2 and 3 I do a systemctl restart mysql.
8. ndb_mgmd SHOW and it shows the same output as above.
9. I goto node 6 and on shell> sudo ndbd --initial
10. It shows:
2021-03-05 21:17:49 [ndbd] INFO -- Angel connected to '172.20.10.50:1186'
2021-03-05 21:17:49 [ndbd] INFO -- Angel allocated nodeid: 6
11. Roughly within a minute I get this output on the mgmd node:
ndb_mgm> Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
12. I get the following error.log, out.log and trace files:
Error log:
==========
GNU nano 4.8 ndb_6_error.log
Current byte-offset of file-pointer is: 3063
Time: Friday 5 March 2021 - 11:26:05
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: tsman.cpp
Error object: TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
Program: ndbd
Pid: 5170
Version: mysql-8.0.23 ndb-8.0.23
Trace file name: ndb_6_trace.log.1
Trace file path: /var/lib/mysql-cluster/ndb_6_trace.log.1 [t1..t1]
***EOM***
>
Time: Friday 5 March 2021 - 16:00:00
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: tsman.cpp
Error object: TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
Program: ndbd
Pid: 1106
Version: mysql-8.0.23 ndb-8.0.23
Trace file name: ndb_6_trace.log.2
Trace file path: /var/lib/mysql-cluster/ndb_6_trace.log.2 [t1..t1]
***EOM***
>
Time: Friday 5 March 2021 - 21:18:01
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: tsman.cpp
Error object: TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
Program: ndbd
Pid: 1493
Version: mysql-8.0.23 ndb-8.0.23
Trace file name: ndb_6_trace.log.3
Trace file path: /var/lib/mysql-cluster/ndb_6_trace.log.3 [t1..t1]
***EOM***