Re: Store C++ STL Vector in MySQL Database
Posted by: Shannon Barber
Date: January 29, 2012 05:09PM

I concur with the "blobs" answer - which means your requirements do not appear to warrant a database and the overhead it brings.

Convert the .csv data into a set of flat binary files.
Somehow you know where to find the data in the .csv files now.
Code that knowledge & logic and you can leverage the OS for caching and pre-fetching of your files.

If the data access is random then with a x64 OS you can memory-map the files into your program's virtual memory space and the OS uses the page-fault mechanism to load your data and I know Windows will anticipate what you will read next and will pre-fetch, I would presume Linux has a similar feature with a sufficiently modern kernel.

If access is linear then aio/asynchronous-IO will allow you to prefetch chunks for processing.

Options: ReplyQuote


Subject
Views
Written By
Posted
3135
A R
January 22, 2012 06:39AM
Re: Store C++ STL Vector in MySQL Database
1417
January 29, 2012 05:09PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.