Hacker News new | past | comments | ask | show | jobs | submit login

Curiosity?



This is silly. If you have a user of your site going through your bots file and specifically going to directories listed as Disallow then you deal with that user. Blocking based on the robots.txt for a directory that doesn't exist anywhere but that file is fine. I did a two bad directory ban, it seemed to work fine.


Curiosity killed the cat, or so they say.


As long as the block lasts not more than a few days, as he says in a nearby comment, I don't think it's much of a problem.


Oh, believe me they only last a few days at most. I don't use an infinite IP blocklist. It has like 100 IP's or so. When one goes in, one must come out. And lets just say there are enough badly behaved bots around that it doesn't take much time for 100 IP's to rotate.


"But satisfaction brought him back."




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: