MySQL Forums
Forum List  »  NDB clusters

Crashed again, but with trace
Posted by: Stephen(Tie) Hu
Date: June 20, 2005 09:17AM

Hi
The node crashed again, but it lasted longer than last time, it took 11 days for this time and with some trace information. Please help.
--------------------------------------------------------------------------------
--MGR Node:
--------------------------------------------------------------------------------
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @209.47.41.10 (Version: 4.1.11, Nodegroup: 0, Master)
id=3 @209.47.41.11 (Version: 4.1.11, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @209.47.41.9 (Version: 4.1.11)

[mysqld(API)] 3 node(s)
id=4 (not connected, accepting connect from sr-10)
id=5 (Version: 4.1.11)
id=6 @209.47.41.10 (Version: 4.1.11)

[mysql@softroute-9 ~]$ ps -ef | grep ndb_mgmd
mysql 11276 1 0 Jun02 ? 00:00:03 ndb_mgmd

ndb_1_cluster.log:
2005-06-18 13:45:46 [MgmSrvr] INFO -- Node 2: Local checkpoint 1851 started. Keep GCI = 2530516 oldest restorable GCI = 2530503
2005-06-18 13:59:08 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected
2005-06-18 13:59:08 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed
2005-06-18 13:59:08 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected
2005-06-18 13:59:08 [MgmSrvr] INFO -- Node 2: Communication to Node 4 closed
2005-06-18 13:59:08 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 000000000000000e.
2005-06-18 13:59:08 [MgmSrvr] INFO -- Mgmt server state: nodeid 6 reserved for ip 209.47.41.10, m_reserved_nodes 000000000000004e.
2005-06-18 13:59:09 [MgmSrvr] INFO -- Node 2: Node 6 Connected
2005-06-18 13:59:09 [MgmSrvr] INFO -- Node 2: Node 6: API version 4.1.11
2005-06-18 13:59:09 [MgmSrvr] INFO -- Node 3: Node 6 Connected
2005-06-18 13:59:09 [MgmSrvr] INFO -- Node 3: Node 6: API version 4.1.11
2005-06-18 13:59:12 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened
2005-06-18 13:59:12 [MgmSrvr] INFO -- Node 2: Communication to Node 4 opened
2005-06-18 14:24:09 [MgmSrvr] INFO -- Node 2: Local checkpoint 1852 started. Keep GCI = 2531578 oldest rest

--------------------------------------------------------------------------------
--Crashed Node(ndbd and mysqld):
--------------------------------------------------------------------------------
myhost-10.err
Number of processes running now: 0
050607 00:28:54 mysqld restarted
050607 0:28:54 [Warning] Changed limits: max_open_files: 1024 max_connections: 300 table_cache: 357
050607 0:28:54 InnoDB: Started; log sequence number 0 137937
/usr/local/mysql/bin/mysqld: ready for connections.
Version: '4.1.11-max' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Edition - Experimental (GPL)
050610 7:43:44 [ERROR] Got error 155 when reading table './ser/subscriber'
050610 7:44:51 [ERROR] /usr/local/mysql/bin/mysqld: Can't open file: 'subscriber.ndb' (errno: 3)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=162
max_connections=300
threads_connected=22
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 660989 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0xafd56bc0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xaf514504, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8136a43
0xd267c8
0x8132381
0x81a4bc5
0x81cb788
0x81490de
0x814b63f
0x814596e
0x8145338
0x81449f8
0xd20341
0xc68fee
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xb349848 = show table status like '#sql-35b4\_578'
thd->thread_id=190505
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
050618 13:51:03 mysqld restarted
050618 13:51:03 [Warning] Changed limits: max_open_files: 1024 max_connections: 300 table_cache: 357
050618 13:51:03 InnoDB: Started; log sequence number 0 137937
/usr/local/mysql/bin/mysqld: ready for connections.
Version: '4.1.11-max' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Edition - Experimental (GPL)

Options: ReplyQuote


Subject
Views
Written By
Posted
3630
June 05, 2005 08:13PM
2654
June 06, 2005 07:43PM
2049
June 07, 2005 07:56AM
2090
June 07, 2005 08:03AM
1946
June 07, 2005 08:57AM
Crashed again, but with trace
3166
June 20, 2005 09:17AM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.