SQL node grows in memory usage
Hello,
We are trying to run cluster of three servers (2 datanodes, 2 sql nodes on same hosts, and management node on separate host, all under RHEL3).
As time goes, mysqld processes are consuming more and more memory. Right after start it is below 10M, shortly comes to 50-80M, and then grows to 200-300M within 24hrs. One time it even consumed more than 1GB memory, so the system slipped in swapping (we have 2GB RAM, 512MB datanode, + system and apache).
I'm not sure if i'm posting this into right forum, but it only began when we moved our database from myisam to cluster; previously with myisam mysqld never occupied more than maybe 80M, being up for several months with same load.
Another change is that previously we were using mysqld-max-5.0.21 (installed from RPMs that were provided) but for clustering we updated it to 5.0.27. We were unable to install provided 5.0.27 RPMs though; so compiled mysql from linux distribution source. Maybe this is problem with release or with our compilation, we didn't try myisam on 5.0.27, to say precisely.
This doesn't seem to be related on query cache, at least, "flush query cache" or "reset query cache" didn't lower mem usage.
Mysqld restart brings mem usage back to low, but then it is starting to increase again.
Below are our current configs. i will be happy if anyone can suggest what to do with this, or what additional info i can provide to help in suggesting.
Thank you in advance.
when mem usage is 327M, reported Threads_created is 12.
our mysqld config:
---------------------------------------
[mysql_cluster]
ndb-connectstring=10.0.0.4,10.0.0.1
[mysqld]
ndbcluster
ft_min_word_len=3
skip-locking
skip-innodb
query_cache_limit=1M
query_cache_size=32M
query_cache_type=1
max_user_connections=52
max_connections=500
safe-show-database
interactive_timeout=10
wait_timeout=10
connect_timeout=10
thread_cache_size=128
key_buffer=16M
join_buffer=1M
max_allowed_packet=16M
table_cache=1024
record_buffer=1M
sort_buffer_size=2M
read_buffer_size=2M
max_connect_errors=10
thread_concurrency=8
myisam_sort_buffer_size=64M
server-id=1
our cluster config:
---------------------------------------
[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=512M
IndexMemory=128M
MaxNoOfAttributes=3000
MaxNoOfTables=384
MaxNoOfOpenFiles=80
NoOfFragmentLogFiles=16
MaxNoOfConcurrentOperations=65536
UndoIndexBuffer=1MB
UndoDataBuffer=8MB
RedoBuffer=16MB
DataDir=/var/lib/mysql-cluster
HeartbeatIntervalDbDb=2500
ArbitrationTimeout=2000
[NDB_MGMD]
Id=14
HostName=10.0.0.4
LogDestination=FILE:filename=/var/lib/mysql-cluster/ndbcluster1.log
DataDir=/var/lib/mysql-cluster
[NDBD]
Id=1
HostName=10.0.0.1
[NDBD]
Id=3
HostName=10.0.0.3
[MYSQLD]
Id=21
HostName=10.0.0.1
[MYSQLD]
Id=23
HostName=10.0.0.3