MySQL Forums
Forum List  »  Performance

Load Data Infile takes a long time
Posted by: Venkatesh Iyer
Date: March 15, 2013 04:44PM

I have an empty table which has about 7 columns. There are indexes on 6 of them. Also the primary key is the combination of all 6 of these columns.

I have also partitioned the table on the date column (ranged per month).

I have a file that has about 20 million records (805M in size). I am trying to execute a LOAD DATA INFILE operation through a php script. It took loads of time, so I split the file into chunks of 100K records. The first set of 100K files took like 20secs per file, however the later ones started slowing down a lot. I also could see that my database machine was getting I/O bounded a lot.

Is there a way that I can speed this whole process? In todays world, ideally 20mill records is not really a big deal right? Looks like I am missing something here. Any pointers would really help.

Options: ReplyQuote


Subject
Views
Written By
Posted
Load Data Infile takes a long time
3509
March 15, 2013 04:44PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.