Hello,
We have three node XtraDB cluster, the cluster was working fine for past few weeks. Today our cluster completely crashed, while checking the logs I found this issue created due to the update. Please find the error below. Any idea why the insertion failure leads to complete server crash.
Thanks in advance for your help.
130409 8:07:21 [Note] WSREP: cleaning up ec4f0d2c-a0f1-11e2-0800-b39cacad59c1 (tcp://10.34.38.140:4567)
len 224; hex a85432b5662b0000792f2fea652b000000fc0d8c622b000000000000000000000000000000000000000000000000000003000000000000000200000000000000010000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000b1dbfe8c8d48a225000000000000000001000000000000008344510700000000deaf2f2c000000000200000000000000010000000000000000fc0d8c622b0000e2cd00000000000060e1117700000000020000000000000058471c60622b0000d8af2f2c000000001800000000000000; asc T2 f+ y// e+ b+ H % DQ /, b+ ` w XG `b+ /, ;
TRANSACTION 3844623, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 1
69 lock struct(s), heap size 14776, 181 row lock(s), undo log entries 6
MySQL thread id 2214011, OS thread handle 0x2b66b20b1940, query id 84509749 10.34.38.153 dbadmin preparing
UPDATE `news` SET `app_status` = 'D', `load_status` = 'D' WHERE ( guid IN (SELECT DISTINCT gallery_id FROM gallery WHERE app_status='A' OR app_status='N') AND feed_id !=21)
130409 8:07:35 InnoDB: Assertion failure in thread 47720073730368 in file btr0pcur.c line 257
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to
http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB:
http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:07:35 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at
http://bugs.percona.com/
key_buffer_size=268435456
read_buffer_size=1048576
max_used_connections=193
max_threads=1250
thread_count=146
connection_count=146
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2837954 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x2b62604aa460
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 2b66b20b10c8 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7c51e5]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x69e894]
/lib64/libpthread.so.0[0x3ecb40eca0]
/lib64/libc.so.6(gsignal+0x35)[0x3eca4302c5]
/lib64/libc.so.6(abort+0x110)[0x3eca431d70]
/usr/sbin/mysqld[0x858dcc]
/usr/sbin/mysqld[0x8062a8]
/usr/sbin/mysqld[0x806bea]
/usr/sbin/mysqld[0x7d7b3d]
/usr/sbin/mysqld(_ZN30subselect_indexsubquery_engine4execEv+0x13d)[0x70bbed]
/usr/sbin/mysqld(_ZN14Item_subselect4execEv+0x4d)[0x70acbd]
/usr/sbin/mysqld(_ZN17Item_in_subselect8val_boolEv+0x20)[0x70b370]
/usr/sbin/mysqld(_ZN17Item_in_optimizer7val_intEv+0x152)[0x6c0552]
/usr/sbin/mysqld(_ZN4Item8val_boolEv+0xac)[0x6ab4bc]
/usr/sbin/mysqld(_ZN13Item_cond_and7val_intEv+0x3a)[0x6c17ca]
/usr/sbin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_ordery15enum_duplicatesbPySB_+0xa80)[0x607bc0]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x25c7)[0x5941a7]
/usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x3b9)[0x5a6599]
/usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x7b)[0x5a9e7b]
/usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x21e)[0x5aa4ae]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x174d)[0x59a02d]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x59a799]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x15f)[0x6356df]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x6358a1]
/lib64/libpthread.so.0[0x3ecb40683d]
/lib64/libc.so.6(clone+0x6d)[0x3eca4d503d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2b62601341e0): is an invalid pointer
Connection ID (thread ID): 2214011
Status: KILL_QUERY