The falcon engine and memory usage

[Vincent Van Den Berghe]

I have a philosophical question about the Falcon engine, the answer of which I'm unable to deduce correctly by looking at the source code.

As I understand it, Falcon is MVCC and keeps uncomitted versions of updated records (and indexes) in memory. But even after a transaction commits, this data stays in memory until it has been written to the datafile (since all updates first appear in the Serial Log, before being written to the datafile by some gopher thread, and the Serial log only contains whatever is intended to show up in the data file)

Falcon uses memory for various types of caches. The allocation of this memory is parameterised, which makes it predictable.
So, the "unpredictable" part of Falcon's memory usage will in part determined by:
(1) The number of concurrent open transactions performing updates (since this increases the number of versions that need to be kept in memory)
(2) The update size of the open transaction (since this will increase the size of the record versions).
(3) (1) + (2)
It's inevitable that this unpredictability occurs for most other storage engines out there. If you have 10 concurrent transactions updating 100Mb blobs, you're bound to use a lot of memory. In Falcon, this memory seems directly proportional to (1) and (2).
But other storage engines I've studied handle such situations by either writing the record versions on disk (and garbage collecting unused versions in a separate thread), or writing the data flagged as "uncommitted" to the log. While this is certainly slower, than Falcon, it makes them much more robust to this out-of-memory vulnerability. They still consume memory (of course) but much much less than Falcon would.

So my questions are:
(1) What can or does Falcon do to reduct this vulnerability, except the usual "Buy more memory"?
(2) Shouldn't such memory consumption have an upper bound, after which data would be written in some kind of USRL (Uncommited Serial Log)?


Vincent