indexing with SQL database

there was a thread here not too long ago about searching /prog/ and i think the general consensus was that downloading and indexing the entire /prog/ and then indexing it was probably the best way of going about it.

but, that got me thinking, what is the best way of putting all this info into a database?
would each word in each thread have it's own entry? or would it be entire posts that got stored?
also, how to deal with all the stupid ascii art and silly tags?

warning: redundant phrase detected near "indexing it"