Eventual Slow Performance and Low Load

[Matt G]

Hi,

I recently converted all my MyIsam tables, which were all running perfectly fine, to InnoDB because of the larger volume of data I was getting. Usually all the queries never take longer than 1-2 seconds and most of them take a small fraction of a second. After updating to InnoDB, everything is running fine and quite fast actually, but consistently if I leave the server on for about 5 hrs, everything goes slower and slower until the point where queries are taking over a couple thousand seconds finish. Through all this, the load remains low as well as the memory usage and nothing gets swapped.

From what I've read, the only suggestion I found to increase performance is to use clustered indexes which I tried doing by adding a primary index (lon, lat, id), but it still slows down eventually. My main tables are users and messages where I need to search through over 500k users by mainly radius, using lat and lon, birthday, as well as a few other fields such as gender. I also join this table by userid a lot and query the user by their id. Strangely, my messages table has about 9M rows and it doesn't seem to get as many slow queries, possibly because I never have to return more than a couple thousand rows.

My server has 24 processors and 48GB of ram, so it's quite fast. The load is only at about 1 when I'm getting this problem. I've listed my config below.

If you have any suggestions to improve performance I would greatly appreciate it! Thank you!


[client]
default-character-set=utf8

[mysql]
default-character-set=utf8

[mysqld]
innodb_data_home_dir='/home/mysql'
log-error=/var/log/mysql/error.log
default-character-set = utf8
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
innodb_file_per_table=1
ft_min_word_len = 3
set-variable = max_connections=2000
max_heap_table_size=256M
tmp_table_size=256M
key_buffer_size=512M
table_cache=1000
sort_buffer_size=512M
read_buffer_size=256M
read_rnd_buffer_size=256M
#myisam_sort_buffer_size = 500M
join_buffer_size = 50M
thread_cache_size = 1000
#wait_timeout = 7200
#connect_timeout = 10
#bulk_insert_buffer_size = 8M
query_cache_limit = 100M
query_cache_size =2048M
query_cache_type = 1
query_prealloc_size = 65536
query_alloc_block_size = 131072
innodb_buffer_pool_size = 20000M
innodb_thread_concurrency = 50
innodb_log_file_size = 256M
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 4M
innodb_additional_mem_pool_size = 20M
large-pages