Name: Anonymous 2010-09-01 14:26
Hi,
I've been experimenting with bypassing basic website security, and I've come across a fairly simple method of spoofing as a botnet.
I configured Firefox to show as a Googlebot by creating general.useragent.override with the simple string "Googlebot/2.1 Compatible: http://www.googlebot.com/bot.html".
Great; it works! Checked to make sure, then enjoyed browsing as a Googlebot.
Now I want to try bypassing sites with robots.txt
Is there any shortcut for this that I can try while still using Googlebot? And if not, is there another crawler with a string I can link to like this one which does not respect robots.txt in the first place?
Thanks.
I've been experimenting with bypassing basic website security, and I've come across a fairly simple method of spoofing as a botnet.
I configured Firefox to show as a Googlebot by creating general.useragent.override with the simple string "Googlebot/2.1 Compatible: http://www.googlebot.com/bot.html".
Great; it works! Checked to make sure, then enjoyed browsing as a Googlebot.
Now I want to try bypassing sites with robots.txt
Is there any shortcut for this that I can try while still using Googlebot? And if not, is there another crawler with a string I can link to like this one which does not respect robots.txt in the first place?
Thanks.