How would I go about this?

I have a ton of grouped pdf files that I need to store in such a way that I can read and search them in the fastest way possible. Write/Annotate/Delete times are not important.

* The pdfs are already grouped by various themes and alphanumerically. This should facilitate searching.

* Average file size is about 2MB.

* There are about 2.5M files

How would I go about this?

>>1
I have never seen a pornographic pdf, much less a collection of those.

>>2

It's erotic literature.

>>3

Oh and thanks for reminding me. What I want to achieve is something similar to literotica.com, albeit with pdf's instead of plain text.

>>1
Use a PDF API for extracting text of PDF.
Build a database with that text.
Extract statistics (use vector model or similar) for terms and themes.

Everything is explained here, without the API thing.
http://nlp.stanford.edu/IR-book/information-retrieval-book.html

>>5
http://nlp.stanford.edu/IR-book/html/htmledition/an-example-information-retrieval-problem-1.html

How do get Antony and Cleopatra and Hamlet from

110100 AND 110111 AND 101111 = 100100

>>6
I don't reccomend using the boolean model because it's shit. Take that only as a reference, any other model should improve on that.

>>6
Explanation:
110100 Is the boolean vector with the documents that have Anthony.
110111 Is the boolean vector with the documents that have Cleopatra.
101111 Is the boolean vector with the documents that have Hamlet.

100100 Is the boolean vector with the documents that have the three terms.

Most models elaborate on this, giving each pair (Document, Term) a value. For each query the value for each document is computed based on that pair's value. On the boolean model those values are absolute (1 = term present, 0 = term not present) but for more advanced models (i.e. non specialized search, like Google) you have to refine this.

>>8
Maybe I'm just being dumb, but I don't see they arrive at

Antony and Cleopatra and Hamlet

Did they refer to the matrix table to get those names? If so, how.

>>9
Those are terms extracted from the document collection. They construct the matrix for the terms they have taken from each document by putting 1 on those cells whose column is a document that contains the term on the row.

>>10
I meant how do they arrive at those terms in the final answer.

>>11
Bitwise and?

>>11
Also, they are not trying to get terms, they are trying to get documents with terms.

>>11
No. Let's try this again.  How do they get

How do they get they Antony and Cleopatra and Hamlet from 100100.

Never mind. I just asked a real computer programmer the same question, and instead of giving my the run around, they gave the answer I was looking for. Funny how this person had no issues comprehending my question.

>>14
On the off chance you are actually asking, and not just trolling and acting dense, the idea is that each indexed bit position represents an entry in some ordered data structure (array, list, tree, whatever), which contains the information needed to resolve the rest of the search data.  When the bit patterns are AND or OR'ed together, depending on the type of search, each set bit represents one of these entries; so you map the bit position to the position in the data structure, dereference the corresponding positions into a result set, and you have your search results.

>>15
Thanks for maturely sharing your real computer programmer's response instead of acting like a whining bitch.

Fucking easy

Install Google Desktop Search
Point it to the folder with all the PDF's.
Wait a day for GDS to index all that shit

PROFIT!

If you want, there's a Python command line tool that you can use to interface with GDS.

>>18
OMG and I was thinking you wanted to do some programming... I am dissapoint.


$ man updatedb; man locate

Well, interfacing with the Python tool is interesting. I played with it one day or two at work, then i became bored.
>>19
I like programming, but is much better when you do something that is needed instead of a toy. In this particular case, just a few tools do the work, and you can even "hide" it building on top of the Python tool, if you really want to.

But writing something that GDS does very well is kinda pointless, IMO.

Still want to play with this?
http://www.sqlite.org/fts3.html
Is a good starting point

< MAN I'M ON A ROLL!

>>5

Thanks for the link. That's what I was looking for!

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　 /＼＿＿_／ヽ　　　　　　　　　　
　　　 (.｀ヽ(｀＞ 、　　　　　　　　　　　　　　　　　　　　　　／''''''　　　'''''':::::　　　　　It's VIP quality
　　 　 `'＜｀ゝr'ﾌ＼　　　　　　　　　　　　　　　　　＋　　|（●）,　　､（●）､.:|　＋　　 http://www.world4ch.org/prog/
　 ⊂ｺ二Lﾌ^´　 ノ, ／⌒)　　　　　　　　　　　　　　　　　|　　,,,ﾉ(､_, )ヽ､,,　.::::|　
　　⊂l二L7_　/　-ゝ-')´　　　　　　　　 　　　　　　　.+　|　　 ｀-=ﾆ=- '　.::::::|　+　.　　　　
　　　　　　 ＼_　　､__,.ｲ＼　　　　　　　　　　　＋　　　　 ＼　　 ｀ﾆﾆ´　 .:::／　　　　+　
　　　　　　　　(T＿_ノ　　 Tヽ　　　　　　　 , -r'⌒!￣ `":::7ヽ.｀- ､　　　./|　　.　
　　　　　　　　　ヽ￢.　　　/　ﾉ`ｰ-､ﾍ<ｰ1´|　 ヽ　| :::::::::::::ﾄ、 ＼　(　　./ヽ　　　　
　　　　　　　　　　＼l__,.／　　 　 　 i　l.ヽ!　|　　　.| ::::::::::::::l ヽ 　 ｀７ｰ.､‐'´ |＼-､

>>21
Too bad SQLite is SLOW AS FUCK.