Re: mysqlimport get very slow for large files

[Aftab Khan]

>key_buffer_size=20GB (was N)

How much RAM do you have on the server?


>KEY `variation_idx` (`variation_id`),
>KEY `subsnp_idx` (`subsnp_id`),
>KEY `sample_idx` (`sample_id`)

Do you require all above indexes? Can we see SELECT queries, so that we may either suggest COMPOSITE index and/or drop redundant index?



>>allele.MYD is now growing at the rate of 20M/s, so in half an hour it should be the size of the txt file (33G).

So it takes longer to rebuild indexes ?

The last setting that can be important is myisam_repair_threads. If you set this variable to a value larger than 1, then MySQL will create multiple indexes at a time. By default, MySQL will only use a single thread in order to do the index creation. This means that on an SMP system that it will only use a single CPU. With this turned on, MySQL will instead create a single thread for each index on the table. This will allow multiple CPU's to be used. One should only turn this on if there are multiple CPU's on a system that you want to use for a repair.



Edited 2 time(s). Last edit at 08/07/2012 08:40AM by Aftab Khan.