Name: Anonymous 2009-01-02 22:38
http://bbsfind.appspot.com/
Sup /prog/
So I was recently approved for a google appspot, and decided to make this. It is a simple webapp which allows you to search for threads based upon title. It uses the subject.txt of the different text boards on dis.4chan.org.
At the moment, it can only search /prog/, but it isn't hard to extend it to all 19 of the current text boards.
Some notes:
* The datastore is using a subjec.txt from just before new year
* At the moment I am scraping the threads locally into CSV format, then uploading them to the datastore.
* Appengine does not currently support threads or cron-like background tasks, which makes automated updating a pain. There are a few hacks around this, though.
(see http://stackoverflow.com/questions/149307/background-tasks-on-appengine for an explanation of the problem)
* You may see some duplicate results, this is because I cancelled a CSV import one time, and had to start again. Ignore for now.
Things todo (high-low priority):
* Automated updating (possibly via http://code.google.com/p/gaeutilities/wiki/Cron). Will probably involve something like selecting the latest "last post time" field, then checking subject.txt for threads created after that time.
* Support for all text boards
* Search posts as well as threads. Not really sure what the best way to do this is though
* Order by title/date in asc/desc order (should be really easy, since all of the functionality is already in Query)
Sup /prog/
So I was recently approved for a google appspot, and decided to make this. It is a simple webapp which allows you to search for threads based upon title. It uses the subject.txt of the different text boards on dis.4chan.org.
At the moment, it can only search /prog/, but it isn't hard to extend it to all 19 of the current text boards.
Some notes:
* The datastore is using a subjec.txt from just before new year
* At the moment I am scraping the threads locally into CSV format, then uploading them to the datastore.
* Appengine does not currently support threads or cron-like background tasks, which makes automated updating a pain. There are a few hacks around this, though.
(see http://stackoverflow.com/questions/149307/background-tasks-on-appengine for an explanation of the problem)
* You may see some duplicate results, this is because I cancelled a CSV import one time, and had to start again. Ignore for now.
Things todo (high-low priority):
* Automated updating (possibly via http://code.google.com/p/gaeutilities/wiki/Cron). Will probably involve something like selecting the latest "last post time" field, then checking subject.txt for threads created after that time.
* Support for all text boards
* Search posts as well as threads. Not really sure what the best way to do this is though
* Order by title/date in asc/desc order (should be really easy, since all of the functionality is already in Query)