MySQL Forums
Forum List  »  Partitioning

Re: Partitioning and multiple threads
Posted by: Rick James
Date: March 31, 2011 10:05PM

A single thread will not do any parallelism. You are using multiple threads, so it makes some sense to split by PARTITION. "Range" partitioning seems reasonable; then you have explicit control over the partition key.

I need a little more info to further discuss this.
* You have 100M rows being INSERTed in a day?
* They need some kind of PRIMARY KEY, but the details of it are not critical? (This may lead to a compound key tailored to the processing.)
* You will be DELETEing _all_ the records eventually? (DELETE is costly; DROP PARTITION is extremely cheap.)
* If you might not DELETE a row immediately, perhaps it could be copied to another table, then logically deleted from the main, big, table?
* Can the INSERTing script(s) pick which PARTITION to INSERT into? (This might be handy.)
* Do the worker threads care what order they process the rows in? (This might lead to "Hash" partitioning instead of Range.)
* Can a worker thread grab 100 items at a time?
* How do you do the INSERTs? Bulk? LOAD DATA?

Options: ReplyQuote

Written By
March 29, 2011 05:26AM
Re: Partitioning and multiple threads
March 31, 2011 10:05PM

Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.