Re: Server just hanging (Copying to tmp table)
There is something DEFINITELY related to open files. I don't see that MySQL is calling ulimit -n xxxx anywhere but I did add it to my script. I actually added "ulimit -n unlimited" and I see my open files limit is 12288 which is the number ulimit puts it to when running that command. I'm getting ahead of myself though. I'll tell you what happened to me this morning (at this point I hadn't added ulimit to the startup script).
Got a page at 6am. My servers were not responding again. I figured it was the server but I decided to take the opportunity to debug. This time I noticed it wasn't hung queries, it's that connections could not be made to the server. Mysqlreport said:
__ Tables ______________________________________________________________
Open 1.23k of 2400 %Cache: 51.42
Opened 6.71k 0.02/s
I checked ulimit -a and it was set to 1024 for "-n" but earlier yesterday I set that to unlimited so something changed it back! (Automated process somehow?)
when I set it to unlimited again on the command line connections started coming through. I found that interesting. Although on one site the server said that I must run "mysqladmin flush-hosts" in order to connect because there were too many concurrent connections (mysql report still listed only 198 out of 300 total connections used)
So, I ran flush-hosts and that allowed that site in and things started running nicer for a few minutes. Then things bogged down again and people couldn't connect (literally about 4 minutes later) I as confused as to why this happened so I checked ulimit again and it was still 12288. I checked mysql report and it was pretty much the same. I logged into the server and flushed tables and things started running smoothly again.
I noticed in sysctl that kern.maxfiles was set to 12288 so I thought this was related to ulimit -n being set to that max. (although I thought at the time ulimit was setting kern.maxfiles but I think that is the other way around) So, I set kern.maxfiles in sysctl.conf to 12288 and rebooted. I found that the server was still running with a ulimit of 1024 unless I manually changed it. So I added "ulimit -n unlimited" to the MySQL server startup script. And I now see what I reported to you at the top.
I think my server hit max files for some reason. I'm going to play around here and see what I can find out.
I know enough to be dangerous. Thanks for being patient!
Subject
Views
Written By
Posted
28556
August 11, 2005 11:25AM
23780
August 12, 2005 09:07AM
19030
August 12, 2005 01:06PM
18524
August 22, 2005 09:46AM
16613
August 24, 2005 03:27PM
13077
August 24, 2005 04:05PM
13782
August 26, 2005 11:02AM
7535
August 26, 2005 11:05AM
6937
August 26, 2005 11:45AM
6696
August 26, 2005 05:46PM
6964
August 26, 2005 10:34PM
6583
August 25, 2005 01:08PM
5967
August 26, 2005 11:07AM
5788
August 27, 2005 09:09AM
6103
August 29, 2005 11:50AM
6024
August 29, 2005 12:04PM
5396
August 29, 2005 12:55PM
5265
August 29, 2005 05:26PM
5235
August 29, 2005 06:34PM
4879
August 29, 2005 08:40PM
4765
August 29, 2005 08:47PM
4629
August 29, 2005 09:00PM
4819
August 30, 2005 10:05AM
4692
August 30, 2005 10:18AM
5219
August 30, 2005 10:20AM
6219
August 29, 2005 08:53PM
5086
August 29, 2005 09:50PM
4934
August 29, 2005 10:01PM
5366
August 30, 2005 08:53AM
4778
August 30, 2005 11:22AM
5318
August 30, 2005 11:33AM
4986
August 30, 2005 11:59AM
24453
August 31, 2005 02:32PM
6634
August 31, 2005 02:49PM
5298
August 31, 2005 02:56PM
5150
August 31, 2005 03:06PM
8424
December 07, 2006 10:30AM
5092
August 31, 2005 03:10PM
4967
August 31, 2005 03:32PM
17810
August 31, 2005 07:15PM
5214
September 01, 2005 03:05AM
7944
September 15, 2005 04:31PM
Re: Server just hanging (Copying to tmp table)
5643
September 01, 2005 04:39AM
4552
September 08, 2005 09:53AM
4947
September 08, 2005 11:25AM
4682
September 26, 2005 11:52AM
4602
September 26, 2005 12:13PM
4531
September 27, 2005 09:55AM
4314
September 27, 2005 10:18AM
4203
September 28, 2005 10:29AM
4361
September 28, 2005 10:34AM
4539
September 28, 2005 10:52AM
4979
September 28, 2005 10:58AM
9793
September 23, 2005 01:56AM
5059
September 23, 2005 07:11AM
5034
October 05, 2005 04:58AM
5513
October 05, 2005 04:52PM
6263
October 05, 2005 11:49PM
5482
October 06, 2005 01:34AM
6831
October 06, 2005 04:57PM
4751
December 13, 2005 10:28AM
4746
December 13, 2005 10:34AM
4644
November 17, 2005 09:39AM
5007
January 08, 2007 12:12PM
4868
December 12, 2005 02:13PM
4703
December 14, 2005 11:43PM
4290
December 15, 2005 08:37AM
4369
December 15, 2005 11:18AM
4283
December 15, 2005 11:27AM
4410
December 15, 2005 11:32AM
4179
December 15, 2005 11:43AM
4118
December 15, 2005 11:43AM
4317
December 15, 2005 11:48AM
4175
December 15, 2005 11:51AM
4582
December 15, 2005 02:14PM
4237
December 15, 2005 02:18PM
4145
December 15, 2005 02:27PM
4594
December 15, 2005 02:43PM
4140
December 15, 2005 08:46AM
5140
January 26, 2006 12:38PM
4262
January 26, 2006 01:48PM
4331
January 26, 2006 01:48PM
4136
January 26, 2006 01:52PM
4270
January 27, 2006 04:46PM
4145
January 29, 2006 11:21AM
4489
January 30, 2006 09:42AM
4520
February 27, 2006 02:17PM
4205
February 27, 2006 02:23PM
4421
March 05, 2006 04:56AM
4250
March 06, 2006 08:09AM
4231
April 05, 2006 01:03PM
4389
September 27, 2006 08:43AM
4347
November 19, 2006 04:42AM
4495
December 14, 2006 05:45PM
4321
February 01, 2007 04:40AM
5003
November 11, 2007 01:24PM
5205
December 20, 2007 10:40AM
6642
January 21, 2008 10:14AM
5981
January 21, 2008 10:53AM
5449
May 19, 2008 09:41AM
5643
February 05, 2008 10:27AM
6225
March 12, 2008 08:37AM
5523
May 19, 2008 03:02AM
6254
November 14, 2008 12:31AM
7298
March 06, 2009 03:36PM