MySQL Forums
Forum List  »  NDB clusters

Re: Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341
Posted by: Pradeep Manickam
Date: March 05, 2021 03:55PM

Hi Tiago,

Thanks for the quick response!

Configuration before change (1 mgm node, 2 data nodes and 2 SQL nodes):

[ndb_mgmd default]
# Directory for MGM node log files
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
#Management Node db1
HostName=172.20.10.50

[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=100M # Memory allocate for data storage


#Directory for Data Node
DataDir=/var/lib/mysql-cluster

[mysqld]
#SQL Node db4
HostName=172.20.10.51

[mysqld]
#SQL Node db5
HostName=172.20.10.52

[ndbd]
#Data Node db2
HostName=172.20.10.54

[ndbd]
#Data Node db3
HostName=172.20.10.53


Configuration after change (1 mgm node, 4 data nodes and 2 SQL nodes):

[ndb_mgmd default]
# Directory for MGM node log files
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
#Management Node db1
HostName=172.20.10.50

[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=100M # Memory allocate for data storage

#Directory for Data Node
DataDir=/var/lib/mysql-cluster

[mysqld]
#SQL Node db4
HostName=172.20.10.51

[mysqld]
#SQL Node db5
HostName=172.20.10.52

[ndbd]
#Data Node db2
HostName=172.20.10.54

[ndbd]
#Data Node db3
HostName=172.20.10.53

[ndbd]
#Data Node db6
HostName=172.20.10.55

[ndbd]
#Data Node db7
HostName=172.20.10.56


I have only added the last two nodes to include 2 more data nodes. I have followed the instructions from this link as detailed there:

https://dev.mysql.com/doc/refman/8.0/en/mysql-cluster-online-add-node-example.html

The version I am using is 8.0.23. It works with 1 management node, 2 data nodes and 2 SQL nodes!

1. I change the config.ini on mgm node to include two more IP addresses for data nodes.
2. On ndb_mgm> 1 STOP
3. Then I start it with sudo ndb_mgmd -f /var/lib/mysql-cluster/config.ini --reload
4. I goto ndb_mgm and SHOW - it shows:

[ndbd(NDB)] 4 node(s)
id=4 @172.20.10.54 (mysql-8.0.23 ndb-8.0.23, Nodegroup: 0, *)
id=5 @172.20.10.53 (mysql-8.0.23 ndb-8.0.23, Nodegroup: 0)
id=6 (not connected, accepting connect from 172.20.10.55)
id=7 (not connected, accepting connect from 172.20.10.56)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.20.10.50 (mysql-8.0.23 ndb-8.0.23)

[mysqld(API)] 2 node(s)
id=2 @172.20.10.51 (mysql-8.0.23 ndb-8.0.23)
id=3 @172.20.10.52 (mysql-8.0.23 ndb-8.0.23)

I do rolling restart by:
5. 4 RESTART - wait for it to restart
6. 5 RESTART - wait for it to restart
7. On SQL nodes 2 and 3 I do a systemctl restart mysql.
8. ndb_mgmd SHOW and it shows the same output as above.
9. I goto node 6 and on shell> sudo ndbd --initial
10. It shows:

2021-03-05 21:17:49 [ndbd] INFO -- Angel connected to '172.20.10.50:1186'
2021-03-05 21:17:49 [ndbd] INFO -- Angel allocated nodeid: 6

11. Roughly within a minute I get this output on the mgmd node:

ndb_mgm> Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

12. I get the following error.log, out.log and trace files:



Edited 1 time(s). Last edit at 03/05/2021 03:59PM by Pradeep Manickam.

Options: ReplyQuote


Subject
Views
Written By
Posted
Re: Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341
247
March 05, 2021 03:55PM


Sorry, only registered users may post in this forum.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.