Re: How to set up an extremely large database

[Rick James]

* By default only 'words' of 3 letters or more are kept. It can be configured to keep more, but at a cost.
* There is a default list of "stop words" (like 'the') that are ignored. This can be changed.
* If a word occurs in more than 50% of documents it is not indexed.
* "Stemming" ('stem' = 'stems' = stemming; etc) is on, and English-centric, I think.
* Selected types of "wild cards" are available, but not everything.
* You can specify that a 'word' _must_ be in the column, or you can say it _should_ be.
* The ranking ("this document more 'relevant' than that") is based on some builtin formulas.

Under the covers there is an inverted index.