Re: Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341

[Pradeep Manickam]

Hi Tiago,

Thanks for the quick response!

Configuration before change (1 mgm node, 2 data nodes and 2 SQL nodes):

[ndb_mgmd default]
# Directory for MGM node log files
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
#Management Node db1
HostName=172.20.10.50

[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=100M # Memory allocate for data storage


#Directory for Data Node
DataDir=/var/lib/mysql-cluster

[mysqld]
#SQL Node db4
HostName=172.20.10.51

[mysqld]
#SQL Node db5
HostName=172.20.10.52

[ndbd]
#Data Node db2
HostName=172.20.10.54

[ndbd]
#Data Node db3
HostName=172.20.10.53


Configuration after change (1 mgm node, 4 data nodes and 2 SQL nodes):

[ndb_mgmd default]
# Directory for MGM node log files
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
#Management Node db1
HostName=172.20.10.50

[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=100M # Memory allocate for data storage

#Directory for Data Node
DataDir=/var/lib/mysql-cluster

[mysqld]
#SQL Node db4
HostName=172.20.10.51

[mysqld]
#SQL Node db5
HostName=172.20.10.52

[ndbd]
#Data Node db2
HostName=172.20.10.54

[ndbd]
#Data Node db3
HostName=172.20.10.53

[ndbd]
#Data Node db6
HostName=172.20.10.55

[ndbd]
#Data Node db7
HostName=172.20.10.56


I have only added the last two nodes to include 2 more data nodes. I have followed the instructions from this link as detailed there:

https://dev.mysql.com/doc/refman/8.0/en/mysql-cluster-online-add-node-example.html

The version I am using is 8.0.23. It works with 1 management node, 2 data nodes and 2 SQL nodes!

1. I change the config.ini on mgm node to include two more IP addresses for data nodes.
2. On ndb_mgm> 1 STOP
3. Then I start it with sudo ndb_mgmd -f /var/lib/mysql-cluster/config.ini --reload
4. I goto ndb_mgm and SHOW - it shows:

[ndbd(NDB)] 4 node(s)
id=4 @172.20.10.54 (mysql-8.0.23 ndb-8.0.23, Nodegroup: 0, *)
id=5 @172.20.10.53 (mysql-8.0.23 ndb-8.0.23, Nodegroup: 0)
id=6 (not connected, accepting connect from 172.20.10.55)
id=7 (not connected, accepting connect from 172.20.10.56)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.20.10.50 (mysql-8.0.23 ndb-8.0.23)

[mysqld(API)] 2 node(s)
id=2 @172.20.10.51 (mysql-8.0.23 ndb-8.0.23)
id=3 @172.20.10.52 (mysql-8.0.23 ndb-8.0.23)

I do rolling restart by:
5. 4 RESTART - wait for it to restart
6. 5 RESTART - wait for it to restart
7. On SQL nodes 2 and 3 I do a systemctl restart mysql.
8. ndb_mgmd SHOW and it shows the same output as above.
9. I goto node 6 and on shell> sudo ndbd --initial
10. It shows:

2021-03-05 21:17:49 [ndbd] INFO -- Angel connected to '172.20.10.50:1186'
2021-03-05 21:17:49 [ndbd] INFO -- Angel allocated nodeid: 6

11. Roughly within a minute I get this output on the mgmd node:

ndb_mgm> Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

12. I get the following error.log, out.log and trace files:



Edited 1 time(s). Last edit at 03/05/2021 03:59PM by Pradeep Manickam.