Re: The falcon engine and memory usage

[Jim Starkey]

It is an historical truism that no matter how much memory you have, it is never enough.

Falcon has a variety of mechanisms to cope with finite memory. Blobs, for example, are not cached in memory but go straight to the serial log or the page cache depending on size. Furthermore, when data and in-memory indexes for a transaction reach predetermined sizes, the data and indexes are flushed to the serial log, taking note of the location in case they need to be reconstructed, and the corresponding memory released (this is particularly nice because it realy doesn't cost anything). The final piece, however, is the "back log", now in development. The back log is a temporary table space for storing fully serialized record chains. The backlog reduces the memory required for an active record to a record number, a bit, and an incomplete tree.

The Falcon design center is where there is sufficient memory to keep all interesting data in memory and back off only when there isn't sufficient memory. We're willing to pay a little extra to but really big transactions in not enough memory to give maximum performance when there is enough.