Re: Problems with system locking

[Ananth Reddy]

Hi Guys,

We are experiencing exactly same problems in production.
I just posted my issue in other recent thread.

We are running MySQL in production for last 10 months and problem started coming now. We have 3-5 tables which are large; raging 30MM rows to 90MM rows in each
All of them or MyISAM tables

When it locks up it only lasts for less than a minute. and even 'SELECT 1' takes 30-60 seconds during this time. This results in application running out of threads and spits out errors to users. It is a web application and we are 24x7

We have instrumented some monitoring to see what is happening during that lock up time.

here is what we have noticed:

Not much IO from vmstats
Load is below average
One mysql thread is maxing out on CPU (25%) == we are suspecting that this thread is some housekeeping thread

System Config:
H/W: Dell 2CPU with internal disks with RAID 5 and hyper-threading
S/W: RedHat 4 with 2.4 kernel
MySQL 4.1.13


If that MySQL thread which is CPU bound is for flushing KEY CACHE dirty buffers, is there a solution to this? We have allocated 1GB for KEY CACHE. I am not sure if it takes that long to walk through 1GB memory. if that is infact what is happening, don't have a solution, then we are in for big trouble as far as scaling is concerned.

Is it possible to find what this one CPU bound mysql thread is doing by sending signal to process to dump trace?

Appreciate any help!

Thanks in Advance

ananth
PS Brian- if you want to talk about it over phone let me know