Interesting Python Scripts

Hey,
I just want to hear some stories about Python and what you did with it. I'm especially interested in if you have ever used mechanize & html5lib (Or something similiar) to write some bots/scrapers etc. for webpages. I actually think forum-bots/spam-bots etc. are annoying but at the same time very interesting, especially when they have got a minimal AI or something comparable to it.

Please don't call me a troll or whatever, that was a real question and I would like to have serious answers and maybe some interesting code-snippets. (Of course I can't force you guys).

Haxus the Great

>>2
Sadly, you didn't care.

http://cairnarvon.rotahall.org/misc/progscrape2.py.html

The original version of 4scrape¹ was in Python. Rewrote the front-end in Haskell because Python is almost SLOW AS FUCK (specifically: embedding a Python interpreter into each Apache daemon made a mess of things, and the Python FastCGI bindings are under-documented nightmares).

The scraper is still in Python though, and just uses urllib2. Never used mechanize or html5lib.

SORRY GUYS HE ASKED AND I COULDN'T RESIST. LOVE, TARO.
                       
^{References: [1] http://suigintou.desudesudesu.org/4scrape/index}

>>4
So you basically scrape with the help of subjects.txt all /prog/ threads and save them into your own local MySQL? Cool, but why?

>>5
[quoute]SORRY GUYS HE ASKED AND I COULDN'T RESIST. LOVE, TARO.[/quote]

So, why are you apologizing?, Sorry I don't seem to get it.

>>6
Because you can then throw them into a fulltext index and search /prog/. Hey Xarn - why not put a small CGI wrapper around that and make it publicaly available?

>>5
Python's FCGI bindings are perfectly acceptable (I know because I've used them in the past). Chances are you just were too dull to understand them.

>>8
Because it's stupidly easy to run it on your own computer.

>>1
I wrote http://dis.4chan.org/read/prog/1185963886/59 and subsequently used it to flood some shitty Serissa-hosted board which is now blisfully dead.

>>6
You get replies from the only two people on /prog/ who actually know how to code and the best you can do is to confuse SQLite with MySQL? No wonder this place is going to shit.

>>9
Try using them for more than a one line script.

>>9
Which bindings did you use? I remember there being like 12 to choose from when I looked into it. I imagine some are better than others.

>>11
Calm down you idiot, it was just a stupid reading mistake, don't take anything and everything grave as if the worlds fate would depend upon it, for real now. If you can't deal with some mistakes (making mistakes is human, and fortunately I am) you should GTFO.

>>14
Toleration of idiotic mistakes leads to the devaluation of the ability not to make them. Or, put simply: kill yourself.

>>15
Or, put simply: kill yourself.

Back to /b/ now.

>>16
At least have some manners.

>>1
(Of course I can't force you guys).
force
forced
THE FORCED INDENTATION OF CODE THREAD OVER

Toleration of idiotic mistakes leads to the devaluation of the ability not to make them
self-aggrandizing bullshit
Oh hi, FrozenVoid

>>19
>>15 isn't FrozenVoid. Although the post does contain a superfluous colon, it isn't followed incorrectly by a capital letter. Plus, all the words are spelled correctly.

>>6,14,16,19
Same worthless faggot trying to make excuses for the fact he doesn't know shit about the differences between database products.

>>21
Na I was the 6 guys, I actually were surprised that I weren't the only one who didn't like FrozenVoid or whatever. So I'll just sage you, have a good day sir :)!

Lain.

>>10
Hello Storlek.

Don't change these.

Name:		Email:
Entire Thread Thread List