MySQL Forums
Forum List  »  InnoDB clusters

Re: MySQL 5.7.18 and memory ...
Posted by: Adam Nelson
Date: September 26, 2017 02:41AM

I was able to preform the upgrade to 5.7.19 this past Friday. Since then the memory usage is more inline with what I would expect. The memory of all nodes is only running around 300GB and rarely goes over that. So it appears the memory leak was the cause of the increased memory usage.. However, the upgrade seems to have introduced something else that while I'm not sure is an issue, really looks like one. New messages have appeared in the mysqld log that were not there before ... these are random samplings from the last 12 - 15 hours ...

Starting right after boot:
2017-09-25T18:49:21.709538Z 7 [Note] Multi-threaded slave statistics for channel 'group_replication_applier': seconds elapsed = 1108; events assigned = 3073; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 1245800 waited (count) when Workers occupied = 2 waited when Workers occupied = 14254900

Few hours later :
2017-09-25T21:11:55.206736Z 7 [Note] Multi-threaded slave statistics for channel 'group_replication_applier': seconds elapsed = 120; events assigned = 13615105; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 73402274100 waited (count) when Workers occupied = 69888 waited when Workers occupied = 313604424700

Few hours later :
2017-09-26T04:06:26.279263Z 7 [Note] Multi-threaded slave statistics for channel 'group_replication_applier': seconds elapsed = 120; events assigned = 43937793; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 213903678300 waited (count) when Workers occupied = 107229 waited when Workers occupied = 525957877500

Few hours later :
2017-09-26T08:04:38.036284Z 7 [Note] Multi-threaded slave statistics for channel 'group_replication_applier': seconds elapsed = 120; events assigned = 83730433; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 300191318600 waited (count) when Workers occupied = 366395 waited when Workers occupied = 1750046504800


The log entry repeats every 120 seconds since boot and my concern is that "events assigned" never decreases ... it hasn't decreased in a single log entry and that "Workers occupied" also always increases. I can't find any documentation on these stats, but just those two stats gives the impression that the slave process is just queuing up and not doing anything. Like it's building to a crash.

Options: ReplyQuote


Subject
Views
Written By
Posted
1209
September 11, 2017 12:44PM
999
September 11, 2017 02:01PM
729
September 13, 2017 07:43AM
Re: MySQL 5.7.18 and memory ...
2662
September 26, 2017 02:41AM
1310
September 11, 2017 08:14PM
954
September 13, 2017 07:41AM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.