Robots.txt

Any reason respecting it?

The only reliable way to protect a page from crawl is to bury it inside of infinitely large set of randomly generated data.

If you don't want your web page to be downloaded, don't put it on the internet.

>>2
That isn't challenging solution.

>>2
If you don't want to see Game Over, don't play Super Mario Bros

>>1
robots.txt is about interacting with search engines that play nice, not "hiding" pages, dipshit.  If you don't want a page found, don't mention it in there.  And if you've got the bandwidth to send an infinitely large amount of random data, how much are you paying per month?  (Assuming you've got a good enough entropy source to refill the pool, or that you mean "not actually random".)

So robots.txt is the retarded way to go.  You could switch content on user-agent, but then again, the crawlers that don't play nice are going to lie anyway.  Or, you could do what anyone with half a fucking brain does:  HTTPS and WWW-Authenticate header.  HTTPS is taxing to do in bulk, so crawlers don't tend to do it as much.  Requiring auth (and the headers not to be snooped due to HTTPS) keeps unwanted eyes out of it, with the usual caveats.

an infinitely large amount of random data
Oy vey!

>>6
ln -s /dev/urandom /var/www/index.html

User-Agent: *
Disallow: /DO%20NOT%20READ/DO%20NOT%20READ/NOT%20CP!!!

>>6
Ok, pseudo-infinitely, you fucking kike