MySQL Forums
Forum List  »  NDB clusters

Need urgently help
Posted by: Torsten Schumann
Date: November 04, 2013 01:18AM

Hello,
i need urgently held. We have a Clustedb with 2 manager, 2 SQL Nodes and 3 Datanodes. I last time Datanode after the other crashes. I restart the node manually. but i don't found the reason.
Here ist the config:
[ndbd default]
NoOfReplicas = 3
DataMemory = 2048M
IndexMemory = 512M
DataDir = D:\mysql_data\log # Verzeichnis der Logdateien für NDBD-Knoten
BackupDataDir = D:\MySQL_Data\ndb_backup

[ndb_mgmd default]
DataDir = D:\mysql_data\log # Verzeichnis der Logdateien für MGM-Knoten

[mysqld default]

[tcp default]

[ndb_mgmd]
NodeId=1
Hostname=10.111.106.27


[ndb_mgmd]
NodeId=2
Hostname=10.111.106.28

[ndbd]
NodeId=3
Hostname=10.111.106.29
datadir=D:\MySQL_Data\ndb_data # Verzeichnis für die Datendateien dieses Knotens


[ndbd]
NodeId=4
Hostname=10.111.106.30
datadir=D:\MySQL_Data\ndb_data # Verzeichnis für die Datendateien dieses Knotens


[ndbd]
NodeId=5
Hostname=10.111.106.31
datadir=D:\MySQL_Data\ndb_data # Verzeichnis für die Datendateien dieses Knotens

[mysqld]
NodeId=51
Hostname=10.111.106.27

[mysqld]
NodeId=52
Hostname=10.111.106.28


[mysqld]
NodeId=53
## Hostname=10.111.106.27

and this are the error messages:

Current byte-offset of file-pointer is: 4143
Time: Saturday 17 August 2013 - 04:12:25
Status: Temporary error, restart node
Message: Node declared dead. See error log for details (Arbitration error)
Error: 2315
Error data: We(5) have been declared dead by 3 (via 3) reason: Hearbeat failure(4)
Error object: QMGR (Line: 3842) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 7356
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.25
***EOM***

Time: Saturday 21 September 2013 - 12:34:58
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 5 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 276) 0x00000002
Program: D:\MySQL\bin\ndbmtd.exe
Pid: 41608 thr: 0
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.1 D:\MySQL_D
Time: Saturday 28 September 2013 - 12:36:28
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 5 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 276) 0x00000002
Program: D:\MySQL\bin\ndbmtd.exe
Pid: 43392 thr: 0
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.2 D:\MySQL_D
Time: Friday 11 October 2013 - 10:39:19
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 5 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 276) 0x00000002
Program: D:\MySQL\bin\ndbmtd.exe
Pid: 45888 thr: 0
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.3 D:\MySQL_Data\
Time: Monday 14 October 2013 - 22:40:20
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 5 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 276) 0x00000002
Program: D:\MySQL\bin\ndbmtd.exe
Pid: 45332 thr: 0
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.4 D:\MySQL_Data\
Time: Monday 21 October 2013 - 01:02:49
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 5 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 276) 0x00000002
Program: D:\MySQL\bin\ndbmtd.exe
Pid: 50040 thr: 0
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.5 D:\MySQL_Data\
Time: Monday 28 October 2013 - 01:01:18
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 5 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 276) 0x00000002
Program: D:\MySQL\bin\ndbmtd.exe
Pid: 51640 thr: 0
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.6 D:\MySQL_Data\
Time: Monday 4 November 2013 - 01:46:11
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 5 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 276) 0x00000002
Program: D:\MySQL\bin\ndbmtd.exe
Pid: 54220 thr: 0
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.7 D:\MySQL_Data\
ta\ndb_da
Time: Sunday 18 March 2012 - 06:55:44
Status: Temporary error, restart node
Message: Conflict when selecting restart type (Internal error, programming error or missing error message, please report a bug)
Error: 2311
Error data: Cluster requires this node to be started with --initial as partial start has been performed and this filesystem is unusable
Error object: DBDIH (Line: 2233) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 2952
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_da
Time: Wednesday 16 May 2012 - 02:22:01
Status: Temporary error, restart node
Message: Node declared dead. See error log for details (Arbitration error)
Error: 2315
Error data: We(5) have been declared dead by 4 (via 4) reason: Hearbeat failure(4)
Error object: QMGR (Line: 3842) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 2064
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.10
***EOM***

Time: Saturday 23 February 2013 - 14:35:58
Status: Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 5883) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 79748
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.11
***EOM***

Time: Saturday 6 April 2013 - 16:54:16
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: .\dbdih\DbdihMain.cpp
Error object: DBDIH (Line: 16695) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 105920
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.12
***EOM***

Time: Thursday 18 April 2013 - 16:00:53
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: .\dbtup\DbtupTrigger.cpp
Error object: DBTUP (Line: 1940) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 105936
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.13
***EOM***

Time: Tuesday 30 April 2013 - 04:12:44
Status: Temporary error, restart node
Message: Node declared dead. See error log for details (Arbitration error)
Error: 2315
Error data: We(5) have been declared dead by 3 (via 3) reason: Hearbeat failure(4)
Error object: QMGR (Line: 3842) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 105704
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.14
***EOM***

Time: Tuesday 30 April 2013 - 07:26:45
Status: Temporary error, restart node
Message: Another node failed during system restart, please investigate error(s) on other node(s) (Restart error)
Error: 2308
Error data: Node 3 disconnected
Error object: QMGR (Line: 3223) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 108412
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.15
***EOM***

Time: Monday 6 May 2013 - 06:53:24
Status: Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 5883) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 112216
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.16
***EOM***

Time: Monday 6 May 2013 - 14:08:33
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: .\dbtux\DbtuxGen.cpp
Error object: DBTUX (Line: 279) 0x00000006
Program: D:\MySQL\bin\ndbd.exe
Pid: 2572
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.17
***EOM***

Time: Monday 6 May 2013 - 16:14:45
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: .\dbtup\DbtupTrigger.cpp
Error object: DBTUP (Line: 1940) 0x00000006
Program: D:\MySQL\bin\ndbd.exe
Pid: 1304
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.18
***EOM***

Time: Friday 10 May 2013 - 10:31:38
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: .\dbtup\DbtupTrigger.cpp
Error object: DBTUP (Line: 1940) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 1744
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.19
***EOM***

Time: Friday 10 May 2013 - 14:02:59
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: .\dbtup\DbtupTrigger.cpp
Error object: DBTUP (Line: 1940) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 3460
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.20
***EOM***

Time: Friday 10 May 2013 - 18:01:14
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: .\dbtup\DbtupTrigger.cpp
Error object: DBTUP (Line: 1940) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 3308
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.21
***EOM***

Time: Friday 10 May 2013 - 23:13:21
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: .\dbtup\DbtupTrigger.cpp
Error object: DBTUP (Line: 1940) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 3252
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.22
***EOM***

Time: Thursday 16 May 2013 - 04:11:33
Status: Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 5883) 0x00000006
Program: D:\MySQL\bin\ndbd.exe
Pid: 5072
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.23
***EOM***

Time: Sunday 19 May 2013 - 04:13:04
Status: Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 5883) 0x00000002
Program: D:\MySQL\bin\ndbd.exe
Pid: 5300
Version: mysql-5.1.56 ndb-7.1.14
Trace: D:\MySQL_Data\ndb_data\ndb_5_trace.log.24
***EOM***

Please help me ;(

Thx T.Schumann

Options: ReplyQuote


Subject
Views
Written By
Posted
Need urgently help
2008
November 04, 2013 01:18AM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.