High CPU Usage with ndbd in VMWare

[Chris Telinde]

Hello,

I have recently put a couple of bi-directionally replicating MySQL clusters into production for Radius. So far things have been working out well, except for the fact that over the past week or so, the CPU usage on each of the data nodes in each cluster has been increasing steadily. Here's my current config.ini for the cluster:

[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2
DataMemory=12288M
IndexMemory=768M
MaxNoOfConcurrentOperations=50000

[ndb_mgmd]
# Management Node 1
# ID is necessary for multiple mgmt nodes
NodeID=1
Hostname=mgmt01
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
# Management Node 2
NodeID=2
Hostname=mgmt02
DataDir=/var/lib/mysql-cluster

[ndbd]
# Options for data node 1
NodeID=3
Hostname=data01
DataDir=/vol1/data/mysql-cluster-data
ServerPort=50501

[ndbd]
# Options for data node 2
NodeID=4
Hostname=data02
DataDir=/vol1/data/mysql-cluster-data
ServerPort=50502

[mysqld]
# Options for SQL node 1
NodeID=5
Hostname=sql01

[mysqld]
# Options for SQL node 2
NodeID=6
Hostname=sql02

[mysqld]
# Options for SQL node 3
NodeID=7
Hostname=sql03

[mysqld]
# Can be used for online backups
# and for using the ndb_desc utility,
# which requires an available api nodeid.

[tcp default]
OverloadLimit=1073741824




The CPU usage right now fluctuates from 50% to 99% on the two data nodes in the cluster. I am curious if anyone has experienced this before. It seems that the more data that is added to the cluster, the harder these data nodes are having to work.

Running the top command on each data node shows ndbd to be the culprit for basically all of the CPU load. I am considering installing and running ndbmtd instead on each server where the data resides to see if this would help, but suggestions would be greatly appreciated. I have seen it elsewhere being stated that running ndbmtd is not suggested within a VMWare environment, which these servers are hosted on.