MySQL Forums
Forum List  »  Data Warehouse

Re: datawarehouse ETL rolls back after 5 GB of data transfer
Posted by: Rick James
Date: May 14, 2010 11:17PM

I prefer to design ETL so that it is interruptable. I make it so that it does things in small chunks (eg, 100 units of work), records where it is, then loops. This avoids huge ROLLBACKs, better handles crashes, etc.

I do see some WHILE loops -- perhaps you could simply move the START/COMMIT to different spots.

No, I did not dig into your code. Are you thinking there is a bug in it?

Options: ReplyQuote


Subject
Views
Written By
Posted
Re: datawarehouse ETL rolls back after 5 GB of data transfer
3452
May 14, 2010 11:17PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.