Xarn

It's been a long time since the last discussion on this topic.

Hello Xarn

>>35
Just write your own, you lazy bum. I did and it was much faster than Xarn's.

>>41
You can test it yourself, but it took ~16 min for a full scrape from scratch, as opposed to ~2 hours with Xarn's scraper¹ (and at that time there were more posts to scrape).
It uses (by default) 16 worker threads and uses the JSON interface (which Xarn might have not been aware of) whenever it can. It also downloads only the missing posts (post selection also works with the JSON interface).
And it has a nice progress bar.

_{1: http://cairnarvon.rotahall.org/2008/12/25/χριστούγεννα/}

>>44
The JSON interface didn't exist when Xarn first wrote /prog/scrape
I heard so too.

the problem with the JSON interface is that there's no way to distinguish a genuine tripcode from a fake one.
That's why my scraper also downloads the HTML versions of posts with tripcodes to verify them.