Re: Error in ndb notes failover
Posted by:
Leo Chan
Date: November 30, 2005 01:43AM
Hi,
I have grep down the log when i disconnected one ndbd node.
From "ndb_4_error.log"
Time: Wednesday 30 November 2005 - 15:44:07
Status: Temporary error, restart node
Message: Arbitrator shutdown, please investigate error(s) on other node(s) (Ar
tration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 3820) 0x0000000a
Program: ./bin/ndbd
Pid: 5059
Trace: /usr/local/mysql/data/ndb_4_trace.log.12
Version: Version 5.0.16
***EOM***
From "ndb_4_trace.log.12"
--------------- Signal ----------------
r.bn: 252 "QMGR", r.proc: 4, r.sigId: 322619 gsn: 164 "CONTINUEB" prio: 0
s.bn: 252 "QMGR", s.proc: 4, s.sigId: 322615 length: 1 trace: 0 #sec: 0 fragInf:
0
H'00000004
--------------- Signal ----------------
r.bn: 252 "QMGR", r.proc: 4, r.sigId: 322613 gsn: 164 "CONTINUEB" prio: 0
s.bn: 252 "QMGR", s.proc: 4, s.sigId: 322610 length: 3 trace: 8 #sec: 0 fragInf:
0
H'00000005 H'00000001 H'00000007
--------------- Signal ----------------
r.bn: 245 "DBTC", r.proc: 4, r.sigId: 316093 gsn: 406 "TCGETOPSIZEREQ" prio: 1
s.bn: 246 "DBDIH", s.proc: 3, s.sigId: 3788714 length: 2 trace: 0 #sec: 0 fragIn
f: 0
H'00000004 H'00f60003
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 4, r.sigId: 322612 gsn: 164 "CONTINUEB" prio: 0
s.bn: 253 "NDBFS", s.proc: 4, s.sigId: 322609 length: 1 trace: 0 #sec: 0 fragInf
: 0
Scanning the memory channel every 10ms
--------------- Signal ----------------
r.bn: 245 "DBTC", r.proc: 4, r.sigId: 316091 gsn: 409 "TIME_SIGNAL" prio: 1
s.bn: 252 "QMGR", s.proc: 4, s.sigId: 316089 length: 1 trace: 0 #sec: 0 fragInf:
0
From "ndb_2_cluster.log"
2005-11-30 15:44:01 [MgmSrvr] WARNING -- Node 4: Node 1 missed heartbeat 2
2005-11-30 15:44:03 [MgmSrvr] WARNING -- Node 4: Node 1 missed heartbeat 3
2005-11-30 15:44:03 [MgmSrvr] INFO -- Node 2: Node 3 Connected
2005-11-30 15:44:04 [MgmSrvr] WARNING -- Node 4: Node 3 missed heartbeat 2
2005-11-30 15:44:04 [MgmSrvr] ALERT -- Node 4: Node 3 Disconnected
2005-11-30 15:44:04 [MgmSrvr] INFO -- Node 4: Communication to Node 3 closed
2005-11-30 15:44:04 [MgmSrvr] ALERT -- Node 4: Network partitioning - arbitra
tion required
2005-11-30 15:44:04 [MgmSrvr] INFO -- Node 4: President restarts arbitration
thread [state=7]
2005-11-30 15:44:04 [MgmSrvr] WARNING -- Node 4: Node 1 missed heartbeat 4
2005-11-30 15:44:04 [MgmSrvr] ALERT -- Node 4: Node 1 declared dead due to mi
ssed heartbeat
2005-11-30 15:44:04 [MgmSrvr] INFO -- Node 4: Lost arbitrator node 1 - proce
ss failure [state=7]
2005-11-30 15:44:04 [MgmSrvr] INFO -- Node 4: Communication to Node 1 closed
2005-11-30 15:44:04 [MgmSrvr] ALERT -- Node 4: Node 1 Disconnected
2005-11-30 15:44:07 [MgmSrvr] INFO -- Node 2: Node 4 Connected
At this case, all mysql api cannot connect to the table that type=ndbcluster and simply return "ERROR 1015 (HY000): Can't lock file (errno: 4009)"