MySQL Forums
Forum List  »  Falcon

The falcon engine and memory usage
Posted by: Vincent Van Den Berghe
Date: February 24, 2008 01:35AM

I have a philosophical question about the Falcon engine, the answer of which I'm unable to deduce correctly by looking at the source code.

As I understand it, Falcon is MVCC and keeps uncomitted versions of updated records (and indexes) in memory. But even after a transaction commits, this data stays in memory until it has been written to the datafile (since all updates first appear in the Serial Log, before being written to the datafile by some gopher thread, and the Serial log only contains whatever is intended to show up in the data file)

Falcon uses memory for various types of caches. The allocation of this memory is parameterised, which makes it predictable.
So, the "unpredictable" part of Falcon's memory usage will in part determined by:
(1) The number of concurrent open transactions performing updates (since this increases the number of versions that need to be kept in memory)
(2) The update size of the open transaction (since this will increase the size of the record versions).
(3) (1) + (2)
It's inevitable that this unpredictability occurs for most other storage engines out there. If you have 10 concurrent transactions updating 100Mb blobs, you're bound to use a lot of memory. In Falcon, this memory seems directly proportional to (1) and (2).
But other storage engines I've studied handle such situations by either writing the record versions on disk (and garbage collecting unused versions in a separate thread), or writing the data flagged as "uncommitted" to the log. While this is certainly slower, than Falcon, it makes them much more robust to this out-of-memory vulnerability. They still consume memory (of course) but much much less than Falcon would.

So my questions are:
(1) What can or does Falcon do to reduct this vulnerability, except the usual "Buy more memory"?
(2) Shouldn't such memory consumption have an upper bound, after which data would be written in some kind of USRL (Uncommited Serial Log)?


Options: ReplyQuote

Written By
The falcon engine and memory usage
February 24, 2008 01:35AM

Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.