MySQL crashing with 'mysqld got signal 11;'

[Iliyan Stoyanov]

Hello fellows. I have a problem with a mysql instance crashing. Here is what I have from the log:

100419 3:01:41 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=2
max_threads=20
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 254079 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0xa6bd4b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xa72ff1e0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x21)[0x8495f41]
/usr/sbin/mysqld(handle_segfault+0x37f)[0x81e77bf]
[0x444400]
/usr/sbin/mysqld(heap_scan+0x2c)[0x836812c]
/usr/sbin/mysqld(_ZN7ha_heap8rnd_nextEPh+0x2f)[0x8365bbf]
/usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x3a)[0x82c3b7a]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x9a)[0x825280a]
/usr/sbin/mysqld[0x82552ac]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0xa69)[0x8261c79]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x137)[0x82639c7]
/usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x146)[0x8264286]
/usr/sbin/mysqld[0x81f20ef]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5c0)[0x81f54f0]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x1fe)[0x81fcb0e]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1183)[0x81fdcb3]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xe1)[0x81fe281]
/usr/sbin/mysqld(handle_one_connection+0x7b1)[0x81ef081]
/lib/libpthread.so.0[0x5aaab5]
/lib/libc.so.6(clone+0x5e)[0x1eadae]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa71bcb00 is an invalid pointer
thd->thread_id=11
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
100419 03:01:41 mysqld_safe Number of processes running now: 0
100419 03:01:41 mysqld_safe mysqld restarted
100419 3:01:41 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
100419 3:01:41 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
100419 3:01:42 InnoDB: Started; log sequence number 0 661063057
100419 3:01:42 [Note] Event Scheduler: Loaded 0 events
100419 3:01:42 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.45-community' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)


The OS is: Fedora release 12 (Constantine)
Linux thunder 2.6.32.11-99.fc12.i686.PAE #1 SMP Mon Apr 5 16:15:03 EDT 2010 i686 i686 i386 GNU/Linux
[18:02]root@thunder:~ # cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.40GHz
stepping : 7
cpu MHz : 2400.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts cid
bogomips : 4745.32
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 32 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.40GHz
stepping : 7
cpu MHz : 2400.000
cache size : 512 KB
physical id : 3
siblings : 2
core id : 0
cpu cores : 1
apicid : 6
initial apicid : 6
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts cid
bogomips : 4744.81
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 32 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.40GHz
stepping : 7
cpu MHz : 2400.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts cid
bogomips : 4744.77
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 32 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.40GHz
stepping : 7
cpu MHz : 2400.000
cache size : 512 KB
physical id : 3
siblings : 2
core id : 0
cpu cores : 1
apicid : 7
initial apicid : 7
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts cid
bogomips : 4744.86
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 32 bits virtual
power management:

[18:02]root@thunder:~ # cat /proc/meminfo
MemTotal: 1544224 kB
MemFree: 632324 kB
Buffers: 56428 kB
Cached: 503648 kB
SwapCached: 0 kB
Active: 421724 kB
Inactive: 312988 kB
Active(anon): 182976 kB
Inactive(anon): 252 kB
Active(file): 238748 kB
Inactive(file): 312736 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 665480 kB
HighFree: 71548 kB
LowTotal: 878744 kB
LowFree: 560776 kB
SwapTotal: 4209008 kB
SwapFree: 4209008 kB
Dirty: 32 kB
Writeback: 0 kB
AnonPages: 174692 kB
Mapped: 40512 kB
Shmem: 8600 kB
Slab: 152012 kB
SReclaimable: 128180 kB
SUnreclaim: 23832 kB
KernelStack: 2112 kB
PageTables: 4988 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4981120 kB
Committed_AS: 935228 kB
VmallocTotal: 122880 kB
VmallocUsed: 26612 kB
VmallocChunk: 62644 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 12280 kB
DirectMap2M: 894976 kB

[18:03]root@thunder:~ # cat /etc/my.cnf
#

[mysqld]
character-set-server = utf8
datadir = /var/lib/mysql
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/lib/mysql/mysql.sock
port = 3306
#general_log = 1
#general_log_file = /var/log/mysql/mysqld.log
log-error = /var/log/mysqld.log
skip-external-locking

max_connections = 20

query_cache_size = 8M
thread_cache_size = 8
query_cache_limit = 1M

open_files_limit = 4000

key_buffer = 128M

read_buffer = 2M
#read_rnd_buffer = 128M
sort_buffer = 4M

query_cache_min_res_unit = 1K
myisam_use_mmap = 1

#optimizer_search_depth = 4
max_heap_table_size = 32M
max_join_size = 256M

innodb_buffer_pool_size = 64M
innodb_additional_mem_pool_size = 8M

innodb_data_file_path = ibdata1:512M;ibdata2:512M;ibdata3:512M;ibdata4:512M;ibdata5:256M:autoextend:max:2G

innodb_log_file_size = 8M
innodb_log_buffer_size = 8M

innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
[client]
socket = /var/lib/mysql/mysql.sock
port = 3306

# End of file

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 11968
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 15
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


The database has around 6Gb of data in it. I am using mostly MyISAM with exception of just a couple of tables on InnoDB. I have used both Fedora Core provided binary and the official binary from MySQL for CentOS5, as well as a build from source code both on FC12 and Crux Linux, which I replaced late yesterday suspecting it's libc of a fault. Any tips what might be causing this are welcome.

If you need more info about the machine that could point you in some direction that I can't see myself just tell me. Thank you all for the support in advance.