MySQL Forums
Forum List  »  NDB clusters

Re: Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341
Posted by: Pradeep Manickam
Date: March 05, 2021 03:52PM

Hi Tiago,

Thanks for the quick response!

Configuration before change (1 mgm node, 2 data nodes and 2 SQL nodes):

[ndb_mgmd default]
# Directory for MGM node log files
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
#Management Node db1
HostName=172.20.10.50

[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=100M # Memory allocate for data storage


#Directory for Data Node
DataDir=/var/lib/mysql-cluster

[mysqld]
#SQL Node db4
HostName=172.20.10.51

[mysqld]
#SQL Node db5
HostName=172.20.10.52

[ndbd]
#Data Node db2
HostName=172.20.10.54

[ndbd]
#Data Node db3
HostName=172.20.10.53


Configuration after change (1 mgm node, 4 data nodes and 2 SQL nodes):

[ndb_mgmd default]
# Directory for MGM node log files
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
#Management Node db1
HostName=172.20.10.50

[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=100M # Memory allocate for data storage

#Directory for Data Node
DataDir=/var/lib/mysql-cluster

[mysqld]
#SQL Node db4
HostName=172.20.10.51

[mysqld]
#SQL Node db5
HostName=172.20.10.52

[ndbd]
#Data Node db2
HostName=172.20.10.54

[ndbd]
#Data Node db3
HostName=172.20.10.53

[ndbd]
#Data Node db6
HostName=172.20.10.55

[ndbd]
#Data Node db7
HostName=172.20.10.56


I have only added the last two nodes to include 2 more data nodes. I have followed the instructions from this link as detailed there:

https://dev.mysql.com/doc/refman/8.0/en/mysql-cluster-online-add-node-example.html

The version I am using is 8.0.23. It works with 1 management node, 2 data nodes and 2 SQL nodes!

1. I change the config.ini on mgm node to include two more IP addresses for data nodes.
2. On ndb_mgm> 1 STOP
3. Then I start it with sudo ndb_mgmd -f /var/lib/mysql-cluster/config.ini --reload
4. I goto ndb_mgm and SHOW - it shows:

[ndbd(NDB)] 4 node(s)
id=4 @172.20.10.54 (mysql-8.0.23 ndb-8.0.23, Nodegroup: 0, *)
id=5 @172.20.10.53 (mysql-8.0.23 ndb-8.0.23, Nodegroup: 0)
id=6 (not connected, accepting connect from 172.20.10.55)
id=7 (not connected, accepting connect from 172.20.10.56)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.20.10.50 (mysql-8.0.23 ndb-8.0.23)

[mysqld(API)] 2 node(s)
id=2 @172.20.10.51 (mysql-8.0.23 ndb-8.0.23)
id=3 @172.20.10.52 (mysql-8.0.23 ndb-8.0.23)

I do rolling restart by:
5. 4 RESTART - wait for it to restart
6. 5 RESTART - wait for it to restart
7. On SQL nodes 2 and 3 I do a systemctl restart mysql.
8. ndb_mgmd SHOW and it shows the same output as above.
9. I goto node 6 and on shell> sudo ndbd --initial
10. It shows:

2021-03-05 21:17:49 [ndbd] INFO -- Angel connected to '172.20.10.50:1186'
2021-03-05 21:17:49 [ndbd] INFO -- Angel allocated nodeid: 6

11. Roughly within a minute I get this output on the mgmd node:

ndb_mgm> Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

12. I get the following error.log, out.log and trace files:

Error log:
==========
GNU nano 4.8 ndb_6_error.log
Current byte-offset of file-pointer is: 3063


Time: Friday 5 March 2021 - 11:26:05
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: tsman.cpp
Error object: TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
Program: ndbd
Pid: 5170
Version: mysql-8.0.23 ndb-8.0.23
Trace file name: ndb_6_trace.log.1
Trace file path: /var/lib/mysql-cluster/ndb_6_trace.log.1 [t1..t1]
***EOM***
>
Time: Friday 5 March 2021 - 16:00:00
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: tsman.cpp
Error object: TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
Program: ndbd
Pid: 1106
Version: mysql-8.0.23 ndb-8.0.23
Trace file name: ndb_6_trace.log.2
Trace file path: /var/lib/mysql-cluster/ndb_6_trace.log.2 [t1..t1]
***EOM***
>
Time: Friday 5 March 2021 - 21:18:01
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: tsman.cpp
Error object: TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
Program: ndbd
Pid: 1493
Version: mysql-8.0.23 ndb-8.0.23
Trace file name: ndb_6_trace.log.3
Trace file path: /var/lib/mysql-cluster/ndb_6_trace.log.3 [t1..t1]
***EOM***

Options: ReplyQuote


Subject
Views
Written By
Posted
Re: Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341
548
March 05, 2021 03:52PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.