NDBD exiting, DynArr256.cpp:683: require((false)) failed (FreeBSD 9)
Posted by: Adam Strohl
Date: October 04, 2012 04:39AM
Date: October 04, 2012 04:39AM
Hello,
I've been trying to bring up a MySQL cluster with 4 servers with the following node configuration under FreeBSD 9:
proxy1 & proxy2 - Management and SQL API nodes
db1 & db2 - NDBD nodes
Everything patches together and connects up, I can even make DBs using the NDB engine. But once I start importing any more than 2-3 tables or inserting more than a few rows both NDB processes exit and I see this on the mgm console:
Node 3: Node shutdown completed. Initiated by signal 15.
Node 4: Node shutdown completed. Initiated by signal 15.
In the ndb_*_out.log files for both NDB nodes this catches my eye:
alloc_chunk(298256 16) -
/home/adams/mysql-cluster-gpl-7.2.8/storage/ndb/src/kernel/vm/DynArr256.cpp:683: require((false)) failed
I've played with all the limits and options I could think of in config.ini, and the amount of data loaded is really small (3 tables with no data at one point as I ignored inserts/rows). The imports I've tried run for only a few seconds.
When just inserting tabledefs it stops at ~4th table:
ERROR 1005 (HY000) at line 184: Can't create table '<DBNAME>.<TABLE>' (errno: 157)
Whe inserting rows and table defs too:
ERROR 1297 (HY000) at line 47: Got temporary error 4010 'Node failure caused abort of transaction' from NDBCLUSTER
They trigger the same error in the log files (see below).
Once NDBD nodes crash like this I can bring them right back up and view the DB (though it is incomplete obviously). I can't even do a "CREATE TABLE test2 (a int) ENGINE=NDB" and that causes the error:
mysql> CREATE TABLE test2 (a int) ENGINE=NDB;
ERROR 1005 (HY000): Can't create table '<DBNAME>.test2' (errno: 157)
(and ndbd exits on both NDBD nodes)
If I drop the DB I was trying to load and create a new one it works again. So I feel like I'm hitting some resource limit but at a total loss as to what. ALL REPORT MEMORY USAGE reports basically no usage even after a crash and when I can't even make that sample test table:
Node 3: Data usage is 0%(718 32K pages of total 262144)
Node 3: Index usage is 0%(167 8K pages of total 131104)
Node 4: Data usage is 0%(718 32K pages of total 262144)
Node 4: Index usage is 0%(167 8K pages of total 131104)
Below is my config and log outputs, any help would be appreciated and I can of course provide more logs etc if needed just let me know.
--------------------- Cluster Config
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @10.37.37.142 (mysql-5.5.27 ndb-7.2.8, Nodegroup: 0)
id=4 @10.37.37.144 (mysql-5.5.27 ndb-7.2.8, Nodegroup: 0, Master)
[ndb_mgmd(MGM)] 2 node(s)
id=1 @10.37.37.134 (mysql-5.5.27 ndb-7.2.8)
id=2 @10.37.37.136 (mysql-5.5.27 ndb-7.2.8)
[mysqld(API)] 2 node(s)
id=5 @10.37.37.134 (mysql-5.5.27 ndb-7.2.8)
id=6 @10.37.37.136 (mysql-5.5.27 ndb-7.2.8)
--------------------- conifg.ini
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
DataMemory=8G # How much memory to allocate for data storage
IndexMemory=1G # How much memory to allocate for index storage
[ndb_mgmd]
hostname=proxy1.dallas.lb # Hostname or IP address of MGM node
datadir=/var/db/mysql-cluster # Directory for MGM node log files
[ndb_mgmd]
hostname=proxy2.dallas.lb # Hostname or IP address of MGM node
datadir=/var/db/mysql-cluster # Directory for MGM node log
[ndbd]
hostname=db1.dallas.lb # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
[ndbd]
hostname=db2.dallas.lb # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
[mysqld]
hostname=proxy1.dallas.lb
[mysqld]
hostname=proxy2.dallas.lb
--------------------- ndb_3_out.log
2012-10-04 06:03:36 [ndbd] INFO -- Angel pid: 87267 started child: 87268
2012-10-04 06:03:36 [ndbd] INFO -- Configuration fetched from 'proxy1.dallas.lb:1186', generation: 1
NDBMT: non-mt
2012-10-04 06:03:36 [ndbd] INFO -- NDB Cluster -- DB node 3
2012-10-04 06:03:36 [ndbd] INFO -- mysql-5.5.27 ndb-7.2.8 --
2012-10-04 06:03:36 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : no numa support
2012-10-04 06:03:36 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 9324Mb initial: 9452Mb
Adding 36Mb to ZONE_LO (1,1151)
Instantiating DBSPJ instanceNo=0
2012-10-04 06:03:36 [ndbd] INFO -- Start initiated (mysql-5.5.27 ndb-7.2.8)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Adding 8156Mb to ZONE_LO (1152,260991)
Adding 1261Mb to ZONE_LO (262145,40328)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
WOPool::init(61, 9)
RWPool::init(22, 14)
blockSz: 800, wpp: 8188 -> 624 (76)
RWPool::init(29, 624)
RWPool::init(42, 16)
RWPool::init(62, 12)
RWPool::init(c2, 18)
RWPool::init(e2, 16)
WOPool::init(41, 8)
RWPool::init(82, 12)
RWPool::init(a2, 54)
WOPool::init(21, 10)
blockSz: 1024, wpp: 8188 -> 816 (28)
RWPool::init(68, 816)
2012-10-04 06:03:39 [ndbd] INFO -- Start phase 0 completed
2012-10-04 06:03:44 [ndbd] INFO -- findNeighbours from: 2100 old (left: 65535 right: 65535) new (4 4)
Dbspj::execSTTOR() inst:0 phase=1
2012-10-04 06:03:44 [ndbd] INFO -- Start phase 1 completed
2012-10-04 06:03:44 [ndbd] INFO -- Start phase 2 completed
2012-10-04 06:03:44 [ndbd] INFO -- Start phase 3 completed
Dbspj::execSTTOR() inst:0 phase=4
2012-10-04 06:03:44 [ndbd] INFO -- Start phase 4 completed
Using 1 fragments per node
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 5 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 6 completed
m_active_buckets.set(0)
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 7 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 8 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 9 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 100 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 101 completed
2012-10-04 06:03:46 [ndbd] INFO -- Node started
alloc_chunk(298256 16) -
/home/adams/mysql-cluster-gpl-7.2.8/storage/ndb/src/kernel/vm/DynArr256.cpp:683: require((false)) failed
2012-10-04 06:05:41 [ndbd] INFO -- Received signal 6. Running error handler.
2012-10-04 06:05:41 [ndbd] INFO -- Signal 6 received; Abort trap: 6
2012-10-04 06:05:41 [ndbd] INFO -- /home/adams/mysql-cluster-gpl-7.2.8/storage/ndb/src/kernel/ndbd.cpp
2012-10-04 06:05:41 [ndbd] INFO -- Error handler signal shutting down system
2012-10-04 06:05:41 [ndbd] INFO -- Error handler shutdown completed - exiting
2012-10-04 06:05:41 [ndbd] ALERT -- Node 3: Forced node shutdown completed. Initiated by signal 6. Caused by error 6000: 'Error OS signal received(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
--------------------- ndb_3_error.log
Time: Thursday 4 October 2012 - 06:05:41
Status: Temporary error, restart node
Message: Error OS signal received (Internal error, programming error or missing error message, please report a bug)
Error: 6000
Error data: Signal 6 received; Abort trap: 6
Error object: /home/adams/mysql-cluster-gpl-7.2.8/storage/ndb/src/kernel/ndbd.cpp
Program: ndbd
Pid: 87268
Version: mysql-5.5.27 ndb-7.2.8
Trace: /usr/local/mysql/data/ndb_3_trace.log.1 [t1..t1]
***EOM***
--------------------- last ~100 lines of ndb_3_trace.log.1
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291036 gsn: 255 "FSCLOSECONF" prio: 0
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291035 length: 1 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291035 gsn: 164 "CONTINUEB" prio: 1
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291032 length: 1 trace: 0 #sec: 0 fragInf: 0
Scanning the memory channel again with no delay
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291034 gsn: 257 "FSCLOSEREQ" prio: 0
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291033 length: 4 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
FilePointer: 173
UserReference: H'00fa0003
Flags: H'00000000, Don't remove file
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291033 gsn: 270 "FSWRITECONF" prio: 0
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291032 length: 1 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291032 gsn: 164 "CONTINUEB" prio: 1
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291029 length: 1 trace: 0 #sec: 0 fragInf: 0
Scanning the memory channel again with no delay
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291031 gsn: 272 "FSWRITEREQ" prio: 0
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291030 length: 8 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
FilePointer: 173
UserReference: H'00fa0003 Operation flag: H'00000011 (Sync, Format=Array of pages)
varIndex: 1
numberOfPages: 1
pageData: H'00000000, H'00000000
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291030 gsn: 259 "FSOPENCONF" prio: 0
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291029 length: 3 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
FilePointer: 173
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291029 gsn: 261 "FSOPENREQ" prio: 0
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291028 length: 11 trace: 0 #sec: 0 fragInf: 0
UserReference: H'00fa0003, userPointer: H'00000000
FileNumber[1-4]: H'00000011 H'ffffffff H'00000000 H'010401ff
FileFlags: H'00000311 Open write only, Sync, Create new file, Truncate existing file
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291028 gsn: 737 "SCHEMA_TRANS_IMPL_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291027 length: 8 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39 opKey: 50331719
requestInfo: 0x3 RequestType: RT_PREPARE opExtra: 0 requestFlags: []
clientRef: 0x10 transId: 0x55e83917
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291027 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 4, s.sigId: -1 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0004 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291026 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291025 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291025 gsn: 737 "SCHEMA_TRANS_IMPL_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291024 length: 8 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39 opKey: 50331718
requestInfo: 0x3 RequestType: RT_PREPARE opExtra: 0 requestFlags: []
clientRef: 0x10 transId: 0x55e83917
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291024 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 4, s.sigId: 247093 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0004 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291023 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291022 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291022 gsn: 654 "INDEX_STAT_IMPL_CONF" prio: 1
s.bn: 258 "DBTUX", s.proc: 3, s.sigId: 291021 length: 2 trace: 0 #sec: 0 fragInf: 0
senderRef: 0x1020003 senderData: 50331717
--------------- Signal ----------------
r.bn: 258 "DBTUX", r.proc: 3, r.sigId: 291021 gsn: 653 "INDEX_STAT_IMPL_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291020 length: 10 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 senderData: 50331717
requestType: RT_START_MON[5] requestFlag: 0x0
indexId: 16 indexVersion: 1 tableId: 12 fragId: 0 fragCount: 2
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291020 gsn: 737 "SCHEMA_TRANS_IMPL_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291019 length: 8 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39 opKey: 50331717
requestInfo: 0x3 RequestType: RT_PREPARE opExtra: 0 requestFlags: []
clientRef: 0x10 transId: 0x55e83917
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291019 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 4, s.sigId: 247090 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0004 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291018 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291017 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39
--
Adam Strohl
A-Team Systems
http://www.ateamsystems.com/
I've been trying to bring up a MySQL cluster with 4 servers with the following node configuration under FreeBSD 9:
proxy1 & proxy2 - Management and SQL API nodes
db1 & db2 - NDBD nodes
Everything patches together and connects up, I can even make DBs using the NDB engine. But once I start importing any more than 2-3 tables or inserting more than a few rows both NDB processes exit and I see this on the mgm console:
Node 3: Node shutdown completed. Initiated by signal 15.
Node 4: Node shutdown completed. Initiated by signal 15.
In the ndb_*_out.log files for both NDB nodes this catches my eye:
alloc_chunk(298256 16) -
/home/adams/mysql-cluster-gpl-7.2.8/storage/ndb/src/kernel/vm/DynArr256.cpp:683: require((false)) failed
I've played with all the limits and options I could think of in config.ini, and the amount of data loaded is really small (3 tables with no data at one point as I ignored inserts/rows). The imports I've tried run for only a few seconds.
When just inserting tabledefs it stops at ~4th table:
ERROR 1005 (HY000) at line 184: Can't create table '<DBNAME>.<TABLE>' (errno: 157)
Whe inserting rows and table defs too:
ERROR 1297 (HY000) at line 47: Got temporary error 4010 'Node failure caused abort of transaction' from NDBCLUSTER
They trigger the same error in the log files (see below).
Once NDBD nodes crash like this I can bring them right back up and view the DB (though it is incomplete obviously). I can't even do a "CREATE TABLE test2 (a int) ENGINE=NDB" and that causes the error:
mysql> CREATE TABLE test2 (a int) ENGINE=NDB;
ERROR 1005 (HY000): Can't create table '<DBNAME>.test2' (errno: 157)
(and ndbd exits on both NDBD nodes)
If I drop the DB I was trying to load and create a new one it works again. So I feel like I'm hitting some resource limit but at a total loss as to what. ALL REPORT MEMORY USAGE reports basically no usage even after a crash and when I can't even make that sample test table:
Node 3: Data usage is 0%(718 32K pages of total 262144)
Node 3: Index usage is 0%(167 8K pages of total 131104)
Node 4: Data usage is 0%(718 32K pages of total 262144)
Node 4: Index usage is 0%(167 8K pages of total 131104)
Below is my config and log outputs, any help would be appreciated and I can of course provide more logs etc if needed just let me know.
--------------------- Cluster Config
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @10.37.37.142 (mysql-5.5.27 ndb-7.2.8, Nodegroup: 0)
id=4 @10.37.37.144 (mysql-5.5.27 ndb-7.2.8, Nodegroup: 0, Master)
[ndb_mgmd(MGM)] 2 node(s)
id=1 @10.37.37.134 (mysql-5.5.27 ndb-7.2.8)
id=2 @10.37.37.136 (mysql-5.5.27 ndb-7.2.8)
[mysqld(API)] 2 node(s)
id=5 @10.37.37.134 (mysql-5.5.27 ndb-7.2.8)
id=6 @10.37.37.136 (mysql-5.5.27 ndb-7.2.8)
--------------------- conifg.ini
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
DataMemory=8G # How much memory to allocate for data storage
IndexMemory=1G # How much memory to allocate for index storage
[ndb_mgmd]
hostname=proxy1.dallas.lb # Hostname or IP address of MGM node
datadir=/var/db/mysql-cluster # Directory for MGM node log files
[ndb_mgmd]
hostname=proxy2.dallas.lb # Hostname or IP address of MGM node
datadir=/var/db/mysql-cluster # Directory for MGM node log
[ndbd]
hostname=db1.dallas.lb # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
[ndbd]
hostname=db2.dallas.lb # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
[mysqld]
hostname=proxy1.dallas.lb
[mysqld]
hostname=proxy2.dallas.lb
--------------------- ndb_3_out.log
2012-10-04 06:03:36 [ndbd] INFO -- Angel pid: 87267 started child: 87268
2012-10-04 06:03:36 [ndbd] INFO -- Configuration fetched from 'proxy1.dallas.lb:1186', generation: 1
NDBMT: non-mt
2012-10-04 06:03:36 [ndbd] INFO -- NDB Cluster -- DB node 3
2012-10-04 06:03:36 [ndbd] INFO -- mysql-5.5.27 ndb-7.2.8 --
2012-10-04 06:03:36 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : no numa support
2012-10-04 06:03:36 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 9324Mb initial: 9452Mb
Adding 36Mb to ZONE_LO (1,1151)
Instantiating DBSPJ instanceNo=0
2012-10-04 06:03:36 [ndbd] INFO -- Start initiated (mysql-5.5.27 ndb-7.2.8)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Adding 8156Mb to ZONE_LO (1152,260991)
Adding 1261Mb to ZONE_LO (262145,40328)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
WOPool::init(61, 9)
RWPool::init(22, 14)
blockSz: 800, wpp: 8188 -> 624 (76)
RWPool::init(29, 624)
RWPool::init(42, 16)
RWPool::init(62, 12)
RWPool::init(c2, 18)
RWPool::init(e2, 16)
WOPool::init(41, 8)
RWPool::init(82, 12)
RWPool::init(a2, 54)
WOPool::init(21, 10)
blockSz: 1024, wpp: 8188 -> 816 (28)
RWPool::init(68, 816)
2012-10-04 06:03:39 [ndbd] INFO -- Start phase 0 completed
2012-10-04 06:03:44 [ndbd] INFO -- findNeighbours from: 2100 old (left: 65535 right: 65535) new (4 4)
Dbspj::execSTTOR() inst:0 phase=1
2012-10-04 06:03:44 [ndbd] INFO -- Start phase 1 completed
2012-10-04 06:03:44 [ndbd] INFO -- Start phase 2 completed
2012-10-04 06:03:44 [ndbd] INFO -- Start phase 3 completed
Dbspj::execSTTOR() inst:0 phase=4
2012-10-04 06:03:44 [ndbd] INFO -- Start phase 4 completed
Using 1 fragments per node
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 5 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 6 completed
m_active_buckets.set(0)
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 7 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 8 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 9 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 100 completed
2012-10-04 06:03:46 [ndbd] INFO -- Start phase 101 completed
2012-10-04 06:03:46 [ndbd] INFO -- Node started
alloc_chunk(298256 16) -
/home/adams/mysql-cluster-gpl-7.2.8/storage/ndb/src/kernel/vm/DynArr256.cpp:683: require((false)) failed
2012-10-04 06:05:41 [ndbd] INFO -- Received signal 6. Running error handler.
2012-10-04 06:05:41 [ndbd] INFO -- Signal 6 received; Abort trap: 6
2012-10-04 06:05:41 [ndbd] INFO -- /home/adams/mysql-cluster-gpl-7.2.8/storage/ndb/src/kernel/ndbd.cpp
2012-10-04 06:05:41 [ndbd] INFO -- Error handler signal shutting down system
2012-10-04 06:05:41 [ndbd] INFO -- Error handler shutdown completed - exiting
2012-10-04 06:05:41 [ndbd] ALERT -- Node 3: Forced node shutdown completed. Initiated by signal 6. Caused by error 6000: 'Error OS signal received(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
--------------------- ndb_3_error.log
Time: Thursday 4 October 2012 - 06:05:41
Status: Temporary error, restart node
Message: Error OS signal received (Internal error, programming error or missing error message, please report a bug)
Error: 6000
Error data: Signal 6 received; Abort trap: 6
Error object: /home/adams/mysql-cluster-gpl-7.2.8/storage/ndb/src/kernel/ndbd.cpp
Program: ndbd
Pid: 87268
Version: mysql-5.5.27 ndb-7.2.8
Trace: /usr/local/mysql/data/ndb_3_trace.log.1 [t1..t1]
***EOM***
--------------------- last ~100 lines of ndb_3_trace.log.1
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291036 gsn: 255 "FSCLOSECONF" prio: 0
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291035 length: 1 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291035 gsn: 164 "CONTINUEB" prio: 1
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291032 length: 1 trace: 0 #sec: 0 fragInf: 0
Scanning the memory channel again with no delay
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291034 gsn: 257 "FSCLOSEREQ" prio: 0
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291033 length: 4 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
FilePointer: 173
UserReference: H'00fa0003
Flags: H'00000000, Don't remove file
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291033 gsn: 270 "FSWRITECONF" prio: 0
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291032 length: 1 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291032 gsn: 164 "CONTINUEB" prio: 1
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291029 length: 1 trace: 0 #sec: 0 fragInf: 0
Scanning the memory channel again with no delay
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291031 gsn: 272 "FSWRITEREQ" prio: 0
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291030 length: 8 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
FilePointer: 173
UserReference: H'00fa0003 Operation flag: H'00000011 (Sync, Format=Array of pages)
varIndex: 1
numberOfPages: 1
pageData: H'00000000, H'00000000
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291030 gsn: 259 "FSOPENCONF" prio: 0
s.bn: 253 "NDBFS", s.proc: 3, s.sigId: 291029 length: 3 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
FilePointer: 173
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 3, r.sigId: 291029 gsn: 261 "FSOPENREQ" prio: 0
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291028 length: 11 trace: 0 #sec: 0 fragInf: 0
UserReference: H'00fa0003, userPointer: H'00000000
FileNumber[1-4]: H'00000011 H'ffffffff H'00000000 H'010401ff
FileFlags: H'00000311 Open write only, Sync, Create new file, Truncate existing file
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291028 gsn: 737 "SCHEMA_TRANS_IMPL_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291027 length: 8 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39 opKey: 50331719
requestInfo: 0x3 RequestType: RT_PREPARE opExtra: 0 requestFlags: []
clientRef: 0x10 transId: 0x55e83917
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291027 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 4, s.sigId: -1 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0004 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291026 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291025 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291025 gsn: 737 "SCHEMA_TRANS_IMPL_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291024 length: 8 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39 opKey: 50331718
requestInfo: 0x3 RequestType: RT_PREPARE opExtra: 0 requestFlags: []
clientRef: 0x10 transId: 0x55e83917
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291024 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 4, s.sigId: 247093 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0004 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291023 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291022 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291022 gsn: 654 "INDEX_STAT_IMPL_CONF" prio: 1
s.bn: 258 "DBTUX", s.proc: 3, s.sigId: 291021 length: 2 trace: 0 #sec: 0 fragInf: 0
senderRef: 0x1020003 senderData: 50331717
--------------- Signal ----------------
r.bn: 258 "DBTUX", r.proc: 3, r.sigId: 291021 gsn: 653 "INDEX_STAT_IMPL_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291020 length: 10 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 senderData: 50331717
requestType: RT_START_MON[5] requestFlag: 0x0
indexId: 16 indexVersion: 1 tableId: 12 fragId: 0 fragCount: 2
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291020 gsn: 737 "SCHEMA_TRANS_IMPL_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291019 length: 8 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39 opKey: 50331717
requestInfo: 0x3 RequestType: RT_PREPARE opExtra: 0 requestFlags: []
clientRef: 0x10 transId: 0x55e83917
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291019 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 4, s.sigId: 247090 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0004 transKey: 39
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 291018 gsn: 738 "SCHEMA_TRANS_IMPL_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 291017 length: 4 trace: 0 #sec: 0 fragInf: 0
senderRef: 0xfa0003 transKey: 39
--
Adam Strohl
A-Team Systems
http://www.ateamsystems.com/
Subject
Views
Written By
Posted
NDBD exiting, DynArr256.cpp:683: require((false)) failed (FreeBSD 9)
2580
October 04, 2012 04:39AM
863
October 08, 2012 06:52AM
927
October 09, 2012 09:02AM
944
October 25, 2012 05:38PM
766
October 26, 2012 12:56AM
Sorry, you can't reply to this topic. It has been closed.
Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.