SECRET FILE DISCOVERED

http://dis.4chan.org/prog/subject.txt

>>19
Actually, the only time you need to look at the HTML version of the page is when there's an exclamation mark in the name field AND the user didn't put anything in the mail field. And even then it's possible to just download only the posts you need to confirm.

Also, my scraper archives the whole /prog/ from scratch in ~15 min. Suck that, Xarn!!

>>23
a) I think we didn't.
b) Pretty well, I made sure it handles correctly everything it can find on /prog/.
=> select count(*) from prog_posts where broken = true;
 count
-------
 22013
(1 row)

=> select count(distinct thread) from prog_posts where broken = true;
 count
-------
    15
(1 row)
where broken = for example, a Haxus the Buggy the Clown.