MySQL Forums
Forum List  »  Replication

MySQL master-slave migration
Posted by: Maggie Liu
Date: April 10, 2016 12:50PM

Hi,

We've failed to migrate our mysql master-slave-slave replication system from old versions to 5.7.

The plan was to move all 6 servers together to 5.7. The whole replication system has 3 layers. One master A running 5.1 with one slave B running 5.5.
Master B has 7 slaves with one 5.1, one 5.5 and five 5.6. It has been running fine with no issues for years.

Then we built brand new servers with 5.7 GA installed, copied all MyIASM tables (*.frm,*.MY*) from existing systems with read lock,export/import INNODB tables,
and run mysql_upgrade on the new systems. Then we set up replication in new system and tested by inserting some data from master server A. It was all working fine.

But the issue happened when we did cutover. We swapped the IP address between old servers and new servers and reset the slaves to point to new master DB. All slaves started properly.

However When the master Server A (5.7) started to get traffic, all 5.7 slaves under Master B start to crash. We can't even keep mysqld up and running. It keeps crashing and crashing until we killed it and turn off slave start.

The strange thing is that replication from master A to Master B is working fine with same version of 5.7 on it. Only 5.7 slaves for Master B are crashing.

The two slaves running 5.6 are actually working fine with 5.7 as master. These two slaves were not touched, We just reset the slave to connect to new master.

Initially we thought INNODB engine was having issues. So we removed iblogfiles and ibdata1 and imported all INNODB tables, still the same issue.

GTID was not enabled.

Do you see what are missed in this migration process? Is it because of the compatibility issue when we copy old version of mysql tables from 5.1 to 5.7?

Is mysql_upgrade causing the data issue? But from mysql_upgrade log, it is showing all successful.


Since we have huge amount of MyISAM data, there's no easy way to migrate other than copying the files.

Could you guys help to see if we missed some major steps here?

The errors we got when crash happened are:

2016-04-10T02:33:02.609592Z 2 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2016-04-10T02:33:02.609637Z 1 [Warning] Slave SQL for channel '': If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0
2016-04-10T02:33:02.609683Z 1 [Note] Slave SQL thread for channel '' initialized, starting replication in log 'dbmcdr-eq-mysql-bin.000009' at position 6110937, relay log './dbscdr2-eq-relay-bin.000002' position: 6110973
2016-04-10T02:33:02.610110Z 2 [Note] Slave I/O thread for channel '': connected to master 'dbscdr2@192.168.130.244:3306',replication started in log 'dbmcdr-eq-mysql-bin.000009' at position 6431210
2016-04-10T02:33:02.612309Z 0 [Note] Event Scheduler: Loaded 0 events
2016-04-10T02:33:02.612455Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.10-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
02:33:02 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=94489280512
read_buffer_size=131072
max_used_connections=0
max_threads=384
thread_count=2
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 92427253 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f93880008c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa9df3157e0 thread_stack 0x40000







/usr/sbin/mysqld(my_print_stacktrace+0x35)[0xef4375]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7a7574]
/lib64/libpthread.so.0(+0xf790)[0x7fa9df080790]
/usr/sbin/mysqld(_ZNK9table_def15compatible_withEP3THDP14Relay_log_infoP5TABLEPS5_+0x1cf)[0xea470f]
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x90f)[0xe7d6ef]
/usr/sbin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x74)[0xe7fcf4]
/usr/sbin/mysqld(_Z26apply_event_and_update_posPP9Log_eventP3THDP14Relay_log_info+0x1dc)[0xeb7cdc]
/usr/sbin/mysqld[0xec23b9]
/usr/sbin/mysqld(handle_slave_sql+0x1041)[0xec4321]
/usr/sbin/mysqld(pfs_spawn_thread+0x174)[0x11e85f4]
/lib64/libpthread.so.0(+0x7a51)[0x7fa9df078a51]
/lib64/libc.so.6(clone+0x6d)[0x7fa9ddbd693d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
160410 02:33:02 mysqld_safe Number of processes running now: 0
160410 02:33:02 mysqld_safe mysqld restarted


Thank you in advance.

Options: ReplyQuote


Subject
Views
Written By
Posted
MySQL master-slave migration
2592
April 10, 2016 12:50PM
971
April 10, 2016 02:50PM
761
April 10, 2016 05:37PM
800
April 10, 2016 10:48PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.