MySQL Forums
Forum List  »  NDB clusters

Memory Allocation Issues
Posted by: Adam D
Date: July 10, 2005 07:21PM

I am having serious issues with my 4 datanode cluster at the moment. I had it all working fine, dumped a whole heap of data on it for testing (when it had the default 100 odd mb IndexMemory and DataMemory settings) All fine. I did a clean shutdown of the cluster.

The OS of all machines is FreeBSD 5.4 using the precompiled FreeBSD Max install. Initial install of the FreeBSD systems had 256mb of memory. After playing around a bit I installed 2gb of memory on 4 of the machines, and left 256 in the management machine.

After I did the dump of a few ndbcluster tables into the cluster - I cannot start the ndbd datanodes at all. I get memory allocation errors, Here is an example of the memory status on datanode 1;
c1s1# vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad4 in sy cs us sy id
0 0 0 35904 1364028 7 0 0 0 10 0 0 416 83 544 0 0 100


I run the /usr/local/mysql/bin/ndbd process and the clusterlog on the mgt server shows;
2005-07-11 11:06:53 [MgmSrvr] INFO -- NDB Cluster Management Server. Version 4.1.12
2005-07-11 11:06:53 [MgmSrvr] INFO -- Id: 1, Command port: 1186
2005-07-11 11:07:00 [MgmSrvr] INFO -- Mgmt server state: nodeid 10 reserved for ip xx.xx.xx.10, m_reserved_nodes 0000000000000402.
2005-07-11 11:07:01 [MgmSrvr] INFO -- Node 1: Node 10 Connected
2005-07-11 11:07:02 [MgmSrvr] INFO -- Mgmt server state: nodeid 10 freed, m_reserved_nodes 0000000000000002.

The ndbd process just ends, the error log shows;

Date/Time: Monday 11 July 2005 - 11:07:01
Type of error: error
Message: Memory allocation failure
Fault ID: 2327
Problem data: DBTUP could not allocate memory for Page
Object of reference: Requested: 32768x12800 = 419430400 bytes
ProgramName: /usr/local/mysql/bin/ndbd
ProcessID: 10556
TraceFile: /spare/mysql-cluster/ndb_10_trace.log.5
Version 4.1.12
***EOM***

I cant make sense of the tracelog, if you guys need it I can provide it.

By that, it looks to be trying to allocate 419430400 (400mb) of memory, which is obviously available according to vmstat. Current memory configuration is 4x512 regecc.


The mgt node config.ini looks like this;
[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=400M
IndexMemory=100M
MaxNoOfConcurrentOperations=10000
MaxNoOfConcurrentTransactions=10000
MaxNoOfLocalOperations=100000
MaxNoOfAttributes = 10000
MaxNoOfOrderedIndexes=5000
MaxNoOfUniqueHashIndexes=5000
MaxNoOfTables=250

4x data nodes, and 4x mysql apis (currently running on the same machines as the datanodes for testing - however changing this doesnt effect)

I tried tinkering with the datamemory values but even on 400 x 2(replicas) still doesnt work. By that it should allocate 800mb for data, and 200mb for index? 1gb. The only thing I can think of perhaps FreeBSD isnt allocating the memory properly or some other strange bug.

The size of the cluster on disk is 377M ndb_10_fs/
Which is the same give or take a couple of meg on each of the 4 data nodes.

Im kinda stuck here if I cant allocat more memory to the database, as it wont even start now. This is the second time it happened, I blew everything away and started again to have the same thing happen when I populated the db.

I have no idea where to turn here, can anyone provide a direction or something to try. Is it an OS bug? Is it a mysql bug, or am I doing something wrong. Thanks.

Options: ReplyQuote


Subject
Views
Written By
Posted
Memory Allocation Issues
2585
July 10, 2005 07:21PM
1624
July 10, 2005 08:09PM
1699
July 10, 2005 09:18PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.