MySQL Forums
Forum List  »  NDB clusters

Re: What is a best way to load large amount of data?
Posted by: Mikael Ronström
Date: May 04, 2006 01:46PM

Hi,


Mark Malakanov wrote:
> I see. Thank you Adam.
>
> The reason I asked I had problems to load a table
> with 1 mln rows by "ALTER TABLE" and by "INSERT ..
> SELECT" methods.
> Always got errors like "REDO is not enough.

This is a typical error when performing high write load.
You need to:
1) Increase NoOfFragmentLogFiles
2) Increase NoOfDiskPagesToDiskAfterRestartTUP/ACC
To calculate amounts check one of my blog entries at
mikaelronstrom.blogspot.com
You might require fairly substantial numbers for number
of fragment log files given that the load on the log at
inserts can be fairly high.

INSERT ... SELECT will most likely suffer from transaction failure
since it is attempted as one transaction.

Rgrds Mikael


> Decrease TimeBetweenLocalCheckpoints or|and
> increase NoOfFragmentLogFiles."
> and even "unknown error".
> I decreased TimeBetweenLocalCheckpoints, increased
> NoOfFragmentLogFiles - but no luck.
>
> My config.ini just in case.
>
>
> NoOfReplicas=1
> DataMemory=2000M
> IndexMemory=200M
> MaxNoOfConcurrentOperations=200000
> NoOfFragmentLogFiles=16
> # 18 - 1Mb; 19 - 2Mb; 20 - 4Mb ...
> TimeBetweenLocalCheckpoints=17
> #TimeBetweenGlobalCheckpoints = 1500
> RedoBuffer=256M
> #NoOfDiskPagesToDiskAfterRestartTUP = 175
> #NoOfDiskPagesToDiskAfterRestartACC = 80
>
>
>
>
>
> # Managment Server
>
> # the IP of THIS SERVER
> HostName=192.168.0.21
>
> # Storage Engines
>
>
>
> I have vorked it around by a procedure with "loop
> fetch; insert; end loop;"
> It worked fine and quite fast.

Mikael Ronstrom
Senior Software Architect, MySQL AB
My blog: http://mikaelronstrom.blogspot.com

Options: ReplyQuote


Subject
Views
Written By
Posted
Re: What is a best way to load large amount of data?
1285
May 04, 2006 01:46PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.