Forced node shutdown completed. Initiated by signal 11. Caused by error 6000

[Willy Ang]

Hi,

I've setup a MySQL 5.0.22 cluster on CentOS 4.3 . In my cluster, I have:

- 2 SQL nodes
- 2 NDB nodes
- 1 MGM node

Only the SQL nodes are on publicly accessible IP addresses. The NDB nodes and MGM nodes are on private IP addresses. The SQL nodes can access the NDB nodes and MGM nodes.

The cluster starts up fine. I'm able to run some queries on the tables through the mysql console. However when I execute a script which runs several SQL commands, one of the NDB nodes will die with the following error:

2006-07-03 15:32:49 [MgmSrvr] ALERT -- Node 2: Forced node shutdown completed. Initiated by signal 11. Caused by error 6000: 'Error OS signal received(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

The following appears in the ndb_2_error.log:

-----------------------------------------------------------------
Time: Monday 3 July 2006 - 15:32:48
Status: Temporary error, restart node
Message: Error OS signal received (Internal error, programming error or missing error message, please report a bug)
Error: 6000
Error data: Signal 11 received; Segmentation fault
Error object: main.cpp
Program: ndbd
Pid: 13899
Trace: /var/lib/mysql-cluster/ndb_2_trace.log.16
Version: Version 5.0.22
-----------------------------------------------------------------

The remaining NDB node becomes the master. When I re-run the same script, this second NDB node also crashes and the entire NDB is down. However I'm able to restart the NDB on each of the NDB nodes with the "ndbd" command.

My config.ini file is:

------------------------------------------------
[NDBD DEFAULT]
NoOfReplicas=2
DataDir=/var/lib/mysql-cluster
MaxNoOfOrderedIndexes=512
MaxNoOfUniqueHashIndexes=512
DataMemory=160M
IndexMemory=72M

# Management Node
[NDB_MGMD]
HostName=192.168.2.2
DataDir=/var/lib/mysql-cluster

# Storage Nodes
# One entry for each node
[NDBD]
HostName=192.168.2.5
[NDBD]
HostName=192.168.2.6

# SQL Nodes
# One entry for each node
[MYSQLD]
HostName=123.123.1.1 # actual external IP address changed
[MYSQLD]
HostName=123.123.1.2 # actual external IP address changed
------------------------------------------------


Is anyone encountering this issue? Is this a bug? Appreciate if someone can provide advise on resolving the issue.

Thank you.



Edited 1 time(s). Last edit at 07/05/2006 11:39PM by Willy Ang.