Re: How to speed up the building of huge (>15 mln entries) normalized database?

[Rick James]

Consider letting MySQL do the work. Toss you cleaver hash table and memory watching; instead:

1. build a big table, Pairs, with 2 columns: word_id, sense_id. Use batched inserts or LOAD DATA.

2. Populate your real table via
INSERT INTO ...
SELECT word_id, sense_id, COUNT(*) FROM Pairs GROUP BY word_id, sense_id;

If your pairs start with quantities other than 1, the change the above to have 3 columns and use SUM().