Which config for large concurrent additions
Posted by: Inbox Roel
Date: March 14, 2016 03:10PM

Hi there.

I could use your help in setting up my database structure the most optimal way (future proof).
This to facilitate data scraped from booking sites based on specific customer profiles. After this I would like to build dashboards with pivots etc for insights through mainly php.

Situation:
- 10k customer profiles per month
- through 5 sites
- with about 125 results
- with 20 attributes (pricing etc) per result
That are minimal 125mio pieces of data for this first iteration...

Following iteration could deviate in > # sites and new/partially overlapping customer profiles (I guess 20k).

Restriction:
- one MySQL database so far

I've thought about limitations of columns, rows and tables, and these are the options that I could think of. Maybe you have better suggestions or point me to one below:
1. 1 table with all profiles, than create table per profile and save the data (# sites, # results, # attributes) per row in this one
2. Same as 1. But to save the amounts of rows per profile table, a blob with per site data (# results and # attributes). But since I've never worked with specifically blobs, I don't know how easy/difficult that would be to get the data out of there to analyze.

Options: ReplyQuote


Subject
Written By
Posted
Which config for large concurrent additions
March 14, 2016 03:10PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.