Skip navigation links

MySQL Forums :: Performance :: Load Data Infile takes a long time


Advanced Search

Load Data Infile takes a long time
Posted by: Venkatesh Iyer ()
Date: March 15, 2013 04:44PM

I have an empty table which has about 7 columns. There are indexes on 6 of them. Also the primary key is the combination of all 6 of these columns.

I have also partitioned the table on the date column (ranged per month).

I have a file that has about 20 million records (805M in size). I am trying to execute a LOAD DATA INFILE operation through a php script. It took loads of time, so I split the file into chunks of 100K records. The first set of 100K files took like 20secs per file, however the later ones started slowing down a lot. I also could see that my database machine was getting I/O bounded a lot.

Is there a way that I can speed this whole process? In todays world, ideally 20mill records is not really a big deal right? Looks like I am missing something here. Any pointers would really help.

Options: ReplyQuote


Subject Views Written By Posted
Load Data Infile takes a long time 1118 Venkatesh Iyer 03/15/2013 04:44PM
Re: Load Data Infile takes a long time 483 Rick James 03/16/2013 06:15PM
Re: Load Data Infile takes a long time 484 Venkatesh Iyer 03/17/2013 11:39PM
Re: Load Data Infile takes a long time 493 addision philip 03/17/2013 11:05PM
Re: Load Data Infile takes a long time 474 Øystein Grøvlen 03/18/2013 03:04AM
Re: Load Data Infile takes a long time 445 Rick James 03/19/2013 08:28PM
Re: Load Data Infile takes a long time 424 Venkatesh Iyer 03/20/2013 12:25PM
Re: Load Data Infile takes a long time 419 Rick James 03/20/2013 05:03PM
Re: Load Data Infile takes a long time 373 Venkatesh Iyer 03/26/2013 11:06AM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.