Disk data file format, restictions, and monitoring

[Brian Z]

Can someone explain the inner workings of the disk data files? How is performance/reliability affected by having few/more large/small data files? (eg.: ALTER TABLESPACE ts_01
ADD DATAFILE 'data_02.dat'
INITIAL_SIZE 100G
ENGINE NDB;
)

I can't help but notice the similarities at-a-glance between MySQL 5.1 Cluster disk data databases, and InnoDB files.

Past experience with InnoDB files led us to use the innodb_file_per_table option to get around the continous problems we had with managing InnoDB file sizes (ie. not auto-shrinking). Cluster data files now do not auto grow either; we need to find out how to manage these properly.

For our cluster, we have 4 identical 2x dual-core servers with 4GB ram each, 4 network cards, 76GB local storage, and iSCSI connected data arrays for the rest of the storage (blade systems with separate disk arrays for each node. I understand the reprecussions of taking the data storage out of the local machine...).

We have about 500GB of data spread across about 140 DB's, and 1400-2000 tables. (Our current servers handling this data are 2x single P4s, 4GB ram, 500GB in RAID10 on a single network).

Is there a guideline for creating/sizing/scaling the disk data files?

Is there a guideline for the the undo log files? I suspect our undo file is not enough for our data, but I don't have a baseline to compare with. We have a single undo log file currently created with:
CREATE LOGFILE GROUP lg_1
ADD UNDOFILE 'undo_1.dat'
INITIAL_SIZE 16M
UNDO_BUFFER_SIZE 2M
ENGINE NDB;


The overall plan is to initially load the data onto the local server data files for OLTP (about 140GB local storage with 76GB on each of 4 nodes with 2 replicas), then use the same nodes and dump the data into a second table space (about 500GB iSCSI partition at a time, growing into 5-7TB).

On a related note, having to change all our create table statements to include the table space is not ideal. Is there any way to force a table space to be used for all tables within a database?:
TABLESPACE ts_01 STORAGE DISK
ENGINE NDB;

Any suggestions on a better way to do this? I really like what Cluster is enabling us to do for our business, but data file management seems a bit clunky to me right now.


Brian