Re: Error 23, 1016: Partition with MyISAM
Posted by:
Rob Redmon
Date: April 01, 2009 10:33AM
First, I'm no DBA, nor a professional programmer. I'm a scientist and this is my opinionated perspective. I found that at 90million rows in one table, queries that should be well indexed were randomly slow and deletes were nearly impossible; corrupting the database in two cases and taking hours to complete in other cases. I'm sure some tuning is possible. Our DBA here counseled me to not put more than a few million rows in a table for the sake of 1) maintainability (e.g. deletes), 2) physical storage (we have small SCSI disks that are shared resources, ~75GB each). I also found insert speed falling off; sorry I don't have hard numbers to back that up at the moment.
The purpose of this file catalog is 2 fold:
1) to tell me exactly where each file lives (e.g. disk, tape, remote location),
2) remove redundant (md5sum) entries broadcast by multiple sources,
3) provide inventory summaries at various temporal granularities (hourly measurements, daily, monthly, etc)
I don't really know what data warehousing is but it sounds this is it.