Large CSV Import - Lost Connection
Hello,
I am trying to import very large csv files into a MySQL database I have created. it is the TPC-H dbgen dataset with a scale of 1000. Basically I have a few tables, and the total data size 1.2TB. I am importing roughly 6 csv files, one is close to 500GB is size. With a 50GB dataset, I am able to import using the "load data local infile" command, however with this larger data i keep hitting the error:
ERROR 2013 (HY000): Lost connection to MySQL server during query
I have tried adjust the wait_timeout and connect_timeout with no luck. Is there a size limit that MySQL can work with, or is there a setting i am missing?
This is the timeout settings I tried using:
| Variable_name | Value |
+----------------------------+----------+
| connect_timeout | 28800 |
| delayed_insert_timeout | 300 |
| innodb_lock_wait_timeout | 120 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| slave_net_timeout | 3600 |
| wait_timeout | 28800 |
+----------------------------+----------+
Thanks!
Subject
Views
Written By
Posted
Large CSV Import - Lost Connection
4442
February 12, 2014 11:52AM
1563
February 17, 2014 12:39AM
Sorry, you can't reply to this topic. It has been closed.
Content reproduced on this site is the property of the respective copyright holders.
It is not reviewed in advance by Oracle and does not necessarily represent the opinion
of Oracle or any other party.