word index - is there a better solution?

[Craig Logan]

hi there,

I'm currently working on a system that allows users to search in about 500M documents (+5M every month).

now the standard approach would of course be to create a word index. this would, however, result in a table of about 100B rows, which kinda scares me because of the performance and maintenance implications (for example, file size limits and how we're going to backup this beast).

I have had mySQL tables with several billion rows that worked fine performance wise, but I guess 100B rows could be a different story.

so basically, I am wondering if there's a different approach for this (even though I'm pretty sure there isn't).

thanks.
craig.