MySQL Forums
Forum List  »  InnoDB

Large CSV Import - Lost Connection
Posted by: Devin Pinkston
Date: February 12, 2014 11:52AM

Hello,

I am trying to import very large csv files into a MySQL database I have created. it is the TPC-H dbgen dataset with a scale of 1000. Basically I have a few tables, and the total data size 1.2TB. I am importing roughly 6 csv files, one is close to 500GB is size. With a 50GB dataset, I am able to import using the "load data local infile" command, however with this larger data i keep hitting the error:

ERROR 2013 (HY000): Lost connection to MySQL server during query


I have tried adjust the wait_timeout and connect_timeout with no luck. Is there a size limit that MySQL can work with, or is there a setting i am missing?

This is the timeout settings I tried using:
| Variable_name | Value |
+----------------------------+----------+
| connect_timeout | 28800 |
| delayed_insert_timeout | 300 |
| innodb_lock_wait_timeout | 120 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| slave_net_timeout | 3600 |
| wait_timeout | 28800 |
+----------------------------+----------+


Thanks!

Options: ReplyQuote


Subject
Views
Written By
Posted
Large CSV Import - Lost Connection
4442
February 12, 2014 11:52AM
1563
February 17, 2014 12:39AM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.