Thanks Aftan. Here is error log.
120803 8:52:25 [Note] Slave SQL thread initialized, starting replication in log 'masterdb-bin.041571' at position 7056475, relay log './mysqld-relay-bin.004127' position: 7203286
120803 8:53:24 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=4294967296
read_buffer_size=1048576
max_used_connections=6
max_threads=300
threads_connected=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4811753 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x7f8d98000970
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f91f040ae58 thread_stack 0x40000
/usr/local/mysql/libexec/mysqld(my_print_stacktrace+0x29) [0x83b379]
/usr/local/mysql/libexec/mysqld(handle_segfault+0x3db) [0x5a8afb]
/lib64/libpthread.so.0() [0x314c60f4a0]
/usr/local/mysql/libexec/mysqld() [0x6a3838]
/usr/local/mysql/libexec/mysqld(mysql_load(THD*, sql_exchange*, TABLE_LIST*, List<Item>&, List<Item>&, List<Item>&, enum_duplicates, bool, bool)+0x16c2) [0x6a6542]
/usr/local/mysql/libexec/mysqld(Load_log_event::do_apply_event(st_net*, Relay_log_info const*, bool)+0x70e) [0x66358e]
/usr/local/mysql/libexec/mysqld(Execute_load_log_event::do_apply_event(Relay_log_info const*)+0x3c0) [0x668690]
/usr/local/mysql/libexec/mysqld(apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*)+0xf5) [0x6c9ec5]
/usr/local/mysql/libexec/mysqld(handle_slave_sql+0x6a2) [0x6cab42]
/lib64/libpthread.so.0() [0x314c6077f1]
/lib64/libc.so.6(clone+0x6d) [0x314bee5ccd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x7f8d98103b28 is an invalid pointer
thd->thread_id=33
thd->killed=NOT_KILLED
The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120803 08:53:24 mysqld_safe Number of processes running now: 0
120803 08:53:24 mysqld_safe mysqld restarted
120803 8:53:24 [Warning] 'for replication startup options' is deprecated and will be removed in a future release. Please use ''CHANGE MASTER'' instead.
120803 8:53:24 [Warning] '--log_slow_queries' is deprecated and will be removed in a future release. Please use ''--slow_query_log'/'--slow_query_log_file'' instead.
120803 8:53:24 [Warning] '--log-long-format' is deprecated and will be removed in a future release. Please use '--log-short-format' instead.
120803 8:53:24 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please use '--skip-external-locking' instead.
120803 8:53:24 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=mysqld-bin' to avoid this problem.
120803 8:53:24 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
120803 8:53:25 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 15901917661
120803 8:53:27 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 15901952983
120803 8:53:28 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 74737409, file name masterdb-bin.039574
InnoDB: Last MySQL binlog file position 0 801, file name ./mysqld-bin.000024
120803 8:53:31 InnoDB Plugin 1.0.6 started; log sequence number 15901952983
120803 8:53:31 [Note] Recovering after a crash using mysqld-bin
120803 8:53:31 [Note] Starting crash recovery...
120803 8:53:31 [Note] Crash recovery finished.
120803 8:53:32 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqld-relay-bin' to avoid this problem.
120803 8:53:32 [Note] Event Scheduler: Loaded 0 events
120803 8:53:32 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '5.1.45-log' socket: '/var/tmp/mysql.sock' port: 33306 Source distribution
120803 10:06:06 [Note] /usr/local/mysql/libexec/mysqld: Normal shutdown
120803 10:06:06 [Note] Event Scheduler: Purging the queue. 0 events
120803 10:06:06 InnoDB: Starting shutdown...
120803 10:06:07 InnoDB: Shutdown completed; log sequence number 15901956903
120803 10:06:07 [Note] /usr/local/mysql/libexec/mysqld: Shutdown complete
120803 10:06:07 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
120803 14:06:51 mysqld_safe Starting mysqld daemon with databases from /opt/data/mysql
120803 14:06:51 [Warning] 'for replication startup options' is deprecated and will be removed in a future release. Please use ''CHANGE MASTER'' instead.
120803 14:06:51 [Warning] '--log_slow_queries' is deprecated and will be removed in a future release. Please use ''--slow_query_log'/'--slow_query_log_file'' instead.
120803 14:06:51 [Warning] '--log-long-format' is deprecated and will be removed in a future release. Please use '--log-short-format' instead.
120803 14:06:51 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please use '--skip-external-locking' instead.
120803 14:06:51 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=mysqld-bin' to avoid this problem.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
120803 14:06:52 InnoDB: highest supported file format is Barracuda.
120803 14:06:54 InnoDB Plugin 1.0.6 started; log sequence number 15907410973
120803 14:06:54 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqld-relay-bin' to avoid this problem.
120803 14:06:54 [Note] Event Scheduler: Loaded 0 events
120803 14:06:54 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '5.1.45-log' socket: '/var/tmp/mysql.sock' port: 33306 Source distribution
120803 14:07:24 [Note] Slave SQL thread initialized, starting replication in log 'masterdb-bin.041578' at position 44965585, relay log './mysqld-relay-bin.004152' position: 46014717
120803 14:07:24 [ERROR] Slave SQL: Error 'Table 'tmpdb.genares_image_20120803123005' doesn't exist' running LOAD DATA INFILE on table 'genares_image_20120803123005'. Default database: '', Error_code: 1146
120803 14:07:24 [ERROR] Slave SQL: Error 'Table 'tmpdb.genares_image_20120803123005' doesn't exist' running LOAD DATA INFILE on table 'genares_image_20120803123005'. Default database: ''. Failed executing load from '/opt/tmp/SQL_LOAD-65203-6520-334.info', Error_code: 1146
120803 14:07:24 [Warning] Slave: Table 'tmpdb.genares_image_20120803123005' doesn't exist Error_code: 1146
120803 14:07:24 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'masterdb-bin.041578' position 44965585
120803 14:07:24 [Note] Slave I/O thread: connected to master 'replication@masterdb:33306',replication started in log 'masterdb-bin.041581' at position 73464943
120803 14:16:33 [Note] Slave SQL thread initialized, starting replication in log 'masterdb-bin.041578' at position 44965585, relay log './mysqld-relay-bin.004152' position: 46014717
120803 14:16:33 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=4294967296
read_buffer_size=1048576
max_used_connections=6
max_threads=300
threads_connected=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4811755 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x7fedd4000970
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7ff2288f1e58 thread_stack 0x40000
/usr/local/mysql/libexec/mysqld(my_print_stacktrace+0x29) [0x850c59]
/usr/local/mysql/libexec/mysqld(handle_segfault+0x3db) [0x5b5ceb]
/lib64/libpthread.so.0(+0xf500) [0x7ff228683500]
/usr/local/mysql/libexec/mysqld() [0x6bb283]
/usr/local/mysql/libexec/mysqld(mysql_load(THD*, sql_exchange*, TABLE_LIST*, List<Item>&, List<Item>&, List<Item>&, enum_duplicates, bool, bool)+0x16c2) [0x6bdf52]
/usr/local/mysql/libexec/mysqld(Load_log_event::do_apply_event(st_net*, Relay_log_info const*, bool)+0x70e) [0x670eee]
/usr/local/mysql/libexec/mysqld(Execute_load_log_event::do_apply_event(Relay_log_info const*)+0x3c0) [0x675ff0]
/usr/local/mysql/libexec/mysqld(apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*)+0xf5) [0x6e24f5]
/usr/local/mysql/libexec/mysqld(handle_slave_sql+0x6a2) [0x6e3172]
/lib64/libpthread.so.0(+0x7851) [0x7ff22867b851]
/lib64/libc.so.6(clone+0x6d) [0x7ff2275d56dd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x7fedd4103b38 is an invalid pointer
thd->thread_id=19
thd->killed=NOT_KILLED
The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120803 14:16:33 mysqld_safe Number of processes running now: 0
120803 14:16:33 mysqld_safe mysqld restarted
120803 14:16:33 [Warning] 'for replication startup options' is deprecated and will be removed in a future release. Please use ''CHANGE MASTER'' instead.
120803 14:16:33 [Warning] '--log_slow_queries' is deprecated and will be removed in a future release. Please use ''--slow_query_log'/'--slow_query_log_file'' instead.
120803 14:16:33 [Warning] '--log-long-format' is deprecated and will be removed in a future release. Please use '--log-short-format' instead.
120803 14:16:33 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please use '--skip-external-locking' instead.
120803 14:16:33 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=mysqld-bin' to avoid this problem.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
120803 14:16:34 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
120803 14:16:35 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 74737409, file name masterdb-bin.039574
InnoDB: Last MySQL binlog file position 0 941, file name ./mysqld-bin.000027
120803 14:16:36 InnoDB Plugin 1.0.6 started; log sequence number 15907410983
120803 14:16:36 [Note] Recovering after a crash using mysqld-bin
120803 14:16:36 [Note] Starting crash recovery...
120803 14:16:36 [Note] Crash recovery finished.
120803 14:16:36 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqld-relay-bin' to avoid this problem.
120803 14:16:36 [Note] Event Scheduler: Loaded 0 events
120803 14:16:36 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '5.1.45-log' socket: '/var/tmp/mysql.sock' port: 33306 Source distribution