Re: Forced shutdown of data node immediately after completion of restart

[Mikael Ronström]

The problem you are facing is due to a bug.
This bug happens while you are in a Software Upgrade situation.
The node that is crashing wants to send a message to DBSPJ on a
different node. So the query causing the crash is some sort of
join query that gets pushed down to NDB.

The bug is in the check itself. Line 4879 in
storage/ndb/src/kernel/blocks/dbtc/DbtcMain.cpp contains the line
ndbrequire(refToMain(TBRef) == DBLQH);
It should state at least
ndbrequire(refToMain(TBRef) == DBLQH ||
refToMain(TBRef) == DBSPJ)
The reason for this check is to ensure that we don't send messages to
DBQLQH (a new query thread block) in a node that doesn't support this.

There is no specific error message simply because this is not supposed
to happen, it is a bug.

Your options are the following.
1) Upgrade using a Cluster restart
This will ensure that no DBSPJ queries will happen during the upgrade
of the cluster
2) Upgrade as now, but ensure that no MySQL Server is doing any pushdown
joins while upgrading (there is a configuration parameter in the MySQL
servers to set this. It is ok to handle DBSPJ queries when all nodes
have upgraded.
3) Fix the binary by simply removing the line with ndbrequire and compile a
new binary.
4) Wait for a release that has a fix for this.

Thx for reporting this bug.