NDB Cluster Data nodes crashing with error 2341

[In-Kap Bang]

Hello,

I am having problems with ndb data nodes that keep crashing with no helpful errors. We are running ndb-7.4.17 with 1 management node, 2 data nodes, and 3 API nodes.

The data nodes keep going down without any warnings/errors in the log, making it very difficult to keep the cluster operational. More often than not, the live node crashes while the second node was recovering, causing the cluster shutdown.

Here is one example of what's logged in the error log in the data node:

Time: Friday 12 January 2018 - 16:59:39
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: Message to big in sendSignal
Error object:
Program: ndbd
Pid: 21910
Version: mysql-5.6.38 ndb-7.4.17
Trace: /home/mysql_data/database/ndb_3_trace.log.7 [t1..t1]
***EOM***

Here is the config.ini for the cluster:

[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
DataMemory=26288M # How much memory to allocate for data storage
IndexMemory=1536M # How much memory to allocate for index storage
MaxNoOfConcurrentOperations=300000
MaxNoOfLocalOperations=330000
MaxNoOfAttributes = 10000 # default 1000
MaxNoOfTables = 1024 # default 128
MaxNoOfOrderedIndexes = 4096 # default 128
MaxNoOfUniqueHashIndexes = 1024 # default 64
#NoOfFragmentLogFiles=300
TimeBetweenGlobalCheckpoints=1000
RedoBuffer=128M

ServerPort=50501 # This is to allocate a fixed port through which one node is connected to the other node within the cluster.
# By default, this port is allocated dynamically in such a way as to ensure that no two nodes on the same host
# computer receive the same port number.
# To open specific ports in a firewall to permit communication between data nodes and API nodes (including SQL
# nodes), you can set this parameter to the number of the desired port in an [ndbd] section or (if you need to do
# this for multiple data nodes) the [ndbd default] section of the config.ini file, and then open the port having
# that number for incoming connections from SQL nodes, API nodes, or both.

[tcp default]
# TCP/IP options:
#portnumber=1186 # This the default; however, you can use any
# port that is free for all the hosts in the cluster
# Note: It is recommended that you do not specify the port
# number at all and simply allow the default value to be used
# instead
SendBufferMemory=20M
ReceiveBufferMemory=20M

[ndb_mgmd]
# Management process options:
NodeId=1
hostname=PUTMYSQL-HA-MGR # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster # Directory for MGM node log files

[ndbd]
# Options for data node "PUTMYSQL05" Replacing PUTMYSQL01:
NodeId=2
hostname=PUTMYSQL05 # Hostname or IP address
datadir=/home/mysql_data/database # Directory for this data node's data files

[ndbd]
# Options for data node "PUTMYSQL06" Replacing PUTMYSQL02:
NodeId=3
hostname=PUTMYSQL06 # Hostname or IP address
datadir=/home/mysql_data/database # Directory for this data node's data files

[mysqld]
# SQL node options:
NodeId=4
hostname=PUTMYSQL03 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)

[mysqld]
#SQL node options:
NodeId=5
hostname=PUTMYSQL04


[mysqld]
#SQL node options:
NodeId=6
hostname=PUTMYSQL07

[mysqld]

[mysqld]

[mysqld]



Any help will be greatly appreciated.

Thank you,