MySQL Forums
Forum List  »  InnoDB

mysql got signal 6 and restarted twice in last 2 days
Posted by: Vaibhav Upadhyay
Date: February 22, 2017 04:24AM

Hi All,

using
mysql 5.6.3
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty

restarting automatically daily since last 2 days.

below are the memory variables
key_buffer_size | 33554432
read_buffer_size | 131072
sort_buffer_size | 262144

max connections configures 500



================================================================================= ERRORs
================================================================================

terminate called after throwing an instance of 'std::out_of_range'
what(): vector::_M_range_check
21:16:50 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=181
max_threads=500
thread_count=135
connection_count=135
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 231166 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8da4ac]
/usr/sbin/mysqld(handle_fatal_signal+0x481)[0x670ee1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f0638b7f330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f0637fc0c37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f0637fc4028]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155)[0x7f06388cb535]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e6d6)[0x7f06388c96d6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e703)[0x7f06388c9703]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e922)[0x7f06388c9922]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x67)[0x7f063891b427]
/usr/sbin/mysqld[0xa5599c]
/usr/sbin/mysqld[0xa5736d]
/usr/sbin/mysqld[0xa58590]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f0638b77184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f063808437d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170220 22:16:52 mysqld_safe Number of processes running now: 0
170220 22:16:52 mysqld_safe mysqld restarted
2017-02-20 22:16:52 0 [Note] /usr/sbin/mysqld (mysqld 5.6.35-log) starting as process 10155 ...
2017-02-20 22:16:52 10155 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
2017-02-20 22:16:52 10155 [Warning] The option innodb (skip-innodb) is deprecated and will be removed in a future release
2017-02-20 22:16:52 10155 [Note] Plugin 'FEDERATED' is disabled.
2017-02-20 22:16:52 10155 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-02-20 22:16:52 10155 [Note] InnoDB: The InnoDB memory heap is disabled
2017-02-20 22:16:52 10155 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-02-20 22:16:52 10155 [Note] InnoDB: Memory barrier is not used
2017-02-20 22:16:52 10155 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-02-20 22:16:52 10155 [Note] InnoDB: Using Linux native AIO
2017-02-20 22:16:52 10155 [Note] InnoDB: Not using CPU crc32 instructions
2017-02-20 22:16:52 10155 [Note] InnoDB: Initializing buffer pool, size = 60.0G
2017-02-20 22:16:56 10155 [Note] InnoDB: Completed initialization of buffer pool
2017-02-20 22:16:57 10155 [Note] InnoDB: Highest supported file format is Barracuda.
2017-02-20 22:16:57 10155 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7729273502457
2017-02-20 22:16:57 10155 [Note] InnoDB: Database was not shutdown normally!
2017-02-20 22:16:57 10155 [Note] InnoDB: Starting crash recovery.
2017-02-20 22:16:57 10155 [Note] InnoDB: Reading tablespace information from the .ibd files...
2017-02-20 22:16:59 10155 [Note] InnoDB: Restoring possible half-written data pages
2017-02-20 22:16:59 10155 [Note] InnoDB: from the doublewrite buffer...

InnoDB: Doing recovery: scanned up to log sequence number 7729278745088
InnoDB: Doing recovery: scanned up to log sequence number 7729283987968

InnoDB: Transaction 21233563079 was in the XA prepared state.
InnoDB: Transaction 21233563079 was in the XA prepared state.
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 0 row operations to undo
InnoDB: Trx id counter is 21233563392
2017-02-20 22:17:08 10155 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 205402331, file name mysql-bin.002993
2017-02-20 22:17:13 10155 [Note] InnoDB: 128 rollback segment(s) are active.
InnoDB: Starting in background the rollback of uncommitted transactions
2017-02-20 22:17:13 7fcf68120700 InnoDB: Rollback of non-prepared transactions completed
2017-02-20 22:17:13 10155 [Note] InnoDB: Waiting for purge to start
2017-02-20 22:17:14 10155 [Note] InnoDB: 5.6.35 started; log sequence number 7729660660496
2017-02-20 22:17:14 7fcfa29de700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
2017-02-20 22:17:14 10155 [Note] Recovering after a crash using /var/lib/mysql/mysql-bin
2017-02-20 22:17:15 10155 [Note] Starting crash recovery...
2017-02-20 22:17:15 7fdf7e6f9780 InnoDB: Starting recovery for XA transactions...
2017-02-20 22:17:15 7fdf7e6f9780 InnoDB: Transaction 21233563079 in prepared state after recovery
2017-02-20 22:17:15 7fdf7e6f9780 InnoDB: Transaction contains changes to 4 rows
2017-02-20 22:17:15 7fdf7e6f9780 InnoDB: 1 transactions in prepared state after recovery
2017-02-20 22:17:15 10155 [Note] Found 1 prepared transaction(s) in InnoDB
2017-02-20 22:17:15 10155 [Note] Crash recovery finished.
2017-02-20 22:17:15 10155 [Note] Server hostname (bind-address): '172.22.0.2'; port: 3306
2017-02-20 22:17:15 10155 [Note] - '172.22.0.2' resolves to '172.22.0.2';
2017-02-20 22:17:15 10155 [Note] Server socket created on IP: '172.22.0.2'.
2017-02-20 22:17:15 7fdf7e6f9780 InnoDB: Error: table `tmp`.`#sql25e7_89_0` does not exist in the InnoDB internal


InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html
2017-02-20 22:17:15 10155 [Warning] 'proxies_priv' entry '@ root@river-db1' ignored in --skip-name-resolve mode.
2017-02-20 22:17:15 10155 [Note] Event Scheduler: Loaded 0 events
2017-02-20 22:17:15 10155 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.35-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)
2017-02-20 22:28:48 7fcfa29de700 InnoDB: Buffer pool(s) load completed at 170220 22:28:48
2017-02-21 06:39:57 10155 [Warning] 'proxies_priv' entry '@ root@river-db1' ignored in --skip-name-resolve mode.
2017-02-21 06:39:58 10155 [Warning] 'proxies_priv' entry '@ root@river-db1' ignored in --skip-name-resolve mode.
terminate called after throwing an instance of 'std::out_of_range'
what(): vector::_M_range_check
07:53:59 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=241
max_threads=500
thread_count=140
connection_count=140
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 231166 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8da4ac]
/usr/sbin/mysqld(handle_fatal_signal+0x481)[0x670ee1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fdf7d875330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fdf7ccb6c37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fdf7ccba028]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155)[0x7fdf7d5c1535]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e6d6)[0x7fdf7d5bf6d6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e703)[0x7fdf7d5bf703]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e922)[0x7fdf7d5bf922]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x67)[0x7fdf7d611427]
/usr/sbin/mysqld[0xa5599c]
/usr/sbin/mysqld[0xa5736d]
/usr/sbin/mysqld[0xa58590]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7fdf7d86d184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fdf7cd7a37d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170222 08:54:01 mysqld_safe Number of processes running now: 0
170222 08:54:01 mysqld_safe mysqld restarted
2017-02-22 08:54:02 0 [Note] /usr/sbin/mysqld (mysqld 5.6.35-log) starting as process 22193 ...
2017-02-22 08:54:02 22193 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.

2017-02-22 08:54:02 22193 [Warning] The option innodb (skip-innodb) is deprecated and will be removed in a future release
2017-02-22 08:54:02 22193 [Note] Plugin 'FEDERATED' is disabled.
2017-02-22 08:54:02 22193 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-02-22 08:54:02 22193 [Note] InnoDB: The InnoDB memory heap is disabled
2017-02-22 08:54:02 22193 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-02-22 08:54:02 22193 [Note] InnoDB: Memory barrier is not used
2017-02-22 08:54:02 22193 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-02-22 08:54:02 22193 [Note] InnoDB: Using Linux native AIO
2017-02-22 08:54:02 22193 [Note] InnoDB: Not using CPU crc32 instructions
2017-02-22 08:54:02 22193 [Note] InnoDB: Initializing buffer pool, size = 60.0G
2017-02-22 08:54:05 22193 [Note] InnoDB: Completed initialization of buffer pool
2017-02-22 08:54:06 22193 [Note] InnoDB: Highest supported file format is Barracuda.
2017-02-22 08:54:06 22193 [Note] InnoDB: Log scan progressed past the checkpoint lsn 7967284821554
2017-02-22 08:54:06 22193 [Note] InnoDB: Database was not shutdown normally!
2017-02-22 08:54:06 22193 [Note] InnoDB: Starting crash recovery.
2017-02-22 08:54:06 22193 [Note] InnoDB: Reading tablespace information from the .ibd files...
2017-02-22 08:54:08 22193 [Note] InnoDB: Restoring possible half-written data pages
2017-02-22 08:54:08 22193 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 7967290064384
InnoDB: Doing recovery: scanned up to log sequence number 7967295307264
InnoDB: Doing recovery: scanned up to log sequence number 7967300550144


InnoDB: Doing recovery: scanned up to log sequence number 7967494553639
InnoDB: Transaction 21463047712 was in the XA prepared state.
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 0 row operations to undo
InnoDB: Trx id counter is 21463048192
2017-02-22 08:54:13 22193 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 826678947, file name mysql-bin.003085
2017-02-22 08:54:18 22193 [Note] InnoDB: 128 rollback segment(s) are active.
InnoDB: Starting in background the rollback of uncommitted transactions
2017-02-22 08:54:18 7fc72381b700 InnoDB: Rollback of non-prepared transactions completed
2017-02-22 08:54:18 22193 [Note] InnoDB: Waiting for purge to start
2017-02-22 08:54:18 22193 [Note] InnoDB: 5.6.35 started; log sequence number 7967494553639
2017-02-22 08:54:18 7fc75e0d9700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
2017-02-22 08:54:18 22193 [Note] Recovering after a crash using /var/lib/mysql/mysql-bin
2017-02-22 08:54:23 22193 [Note] Starting crash recovery...
2017-02-22 08:54:23 7fd739df4780 InnoDB: Starting recovery for XA transactions...
2017-02-22 08:54:23 7fd739df4780 InnoDB: Transaction 21463047712 in prepared state after recovery
2017-02-22 08:54:23 7fd739df4780 InnoDB: Transaction contains changes to 5 rows
2017-02-22 08:54:23 7fd739df4780 InnoDB: 1 transactions in prepared state after recovery
2017-02-22 08:54:23 22193 [Note] Found 1 prepared transaction(s) in InnoDB
2017-02-22 08:54:23 22193 [Note] Crash recovery finished.
2017-02-22 08:54:23 22193 [Note] Server hostname (bind-address): '172.22.0.2'; port: 3306
2017-02-22 08:54:23 22193 [Note] - '172.22.0.2' resolves to '172.22.0.2';
2017-02-22 08:54:23 22193 [Note] Server socket created on IP: '172.22.0.2'.
2017-02-22 08:54:23 7fd739df4780 InnoDB: Error: table `tmp`.`#sql27ab_24dd7_0` does not exist in the InnoDB internal


InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html
2017-02-22 08:54:23 7fd739df4780 InnoDB: Error: table `tmp`.`#sql27ab_22176_0` does not exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html
2017-02-22 08:54:23 22193 [Warning] 'proxies_priv' entry '@ root@river-db1' ignored in --skip-name-resolve mode.
2017-02-22 08:54:23 22193 [Note] Event Scheduler: Loaded 0 events
2017-02-22 08:54:23 22193 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.35-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)
2017-02-22 08:54:23 22193 [ERROR] /usr/sbin/mysqld: Table './mysql/proc' is marked as crashed and should be repaired
2017-02-22 08:54:23 22193 [Warning] Checking table: './mysql/proc'
2017-02-22 09:05:54 7fc75e0d9700 InnoDB: Buffer pool(s) load completed at 170222 9:05:54

Options: ReplyQuote




Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.