indexing with SQL database

there was a thread here not too long ago about searching /prog/ and i think the general consensus was that downloading and indexing the entire /prog/ and then indexing it was probably the best way of going about it.

but, that got me thinking, what is the best way of putting all this info into a database?
would each word in each thread have it's own entry? or would it be entire posts that got stored?
also, how to deal with all the stupid ascii art and silly tags?

warning: redundant phrase detected near "indexing it"

>>2
indeed, please disregard when reading

The HTML backend to the /prog/ site is completely regular. It's easy to strip all container elements leaving nothing but code that conveys the post.

Just store the entire posts and use full-text search. The whole thing was only about 8MB last time I looked, IIRC.

MAKE SURE TO INDEX IT TWICE GUYS

u meen index liek in boox?

>>5
http://dis.4chan.org/read/prog/1209656851

Are you GAY?
Are you a NIGGER?
Are you a GAY NIGGER?

If you answered "Yes" to all of the above questions, then GNAA (GAY NIGGER ASSOCIATION OF AMERICA) might be exactly what you've been looking for!

<-- check em dubz