Hello, all! Sorry to keep bugging you with issues.
We just recently upgraded our cluster to 5.0.18 from 4.1.11. Everything was going great until one of the node machines crashed. Now we have a weird problem coming up. Here's an example of part of the problem. Even though I run "purge stale sessions" from the management node, I get the following error when restarting the API node:
-----
root@blah:~# /etc/rc.d/rc.mysql.mysqld start
root@blah:~# Configuration error: Could not alloc node id at x.x.x.57 port
1186: No free node id found for mysqld(API).
060207 15:53:23 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '5.0.18' socket: '/tmp/mysql.sock' port: 3306 Source distribution
-----
The odd part is that it says it can't allocate a node ID, then it says it started and is ready for connections. That's new behavior. Our network monitoring program seems to think the API node is now up, but the management node insists it's still down.
So we tried to go into mysql on the node in question and run a query. This is what we got:
-----
mysql> select * from blah limit 10;
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=268435456
read_buffer_size=1044480
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections =
466543 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x844a770
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbf5fd638, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81799ac
0x4015bbe5
0x8329e22
0x8230c54
0x822f326
0x821ff9e
0x81b9c4f
0x81b3cee
0x81b2f17
0x81b4469
0x81b489a
0x818cfc4
0x819503e
0x818b9f6
0x818b572
0x818a953
0x40155cc4
0x403a1037
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read
http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and
follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x84727a8 = select * from blah limit 10
thd->thread_id=2
The manual page at
http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
ERROR 2013 (HY000): Lost connection to MySQL server during query
-----
I have never seen that before. I have since rebooted the server, so I don't know if trying to run a stack trace will help now or not.
Any chance anyone knows what I can try next? Thank you.