Adding stemming to fulltext TF-IDF score queries

[Jason Wisdom]

Hi everybody,

(originally posted this in the UDF section for some strange reason; apologies to the management of these fine forums)

I am building a text analytics engine, using scoring through TF-IDF method. Simple as this:


SET @keyword='+deficiencies';

SELECT documentid, documentname, ROUND(MATCH(documenttext) AGAINST (CONCAT('-no ', @keyword) IN BOOLEAN MODE), 7) AS score FROM documenttable;

(fulltext index on longtext column 'documenttext')


What I WANT to do is include words like "deficient" "deficiency" etc. into my search. In other words, stemming.

MATCH() sucks with SOUNDEX(), they don't work together at all.
But how can I include stemming into my fulltext scoring statement?