There are a lot of crawlers out there, and many of them are ill-behaved. When Go...

wolfgang42 on July 23, 2022 | parent | context | favorite | on: Writing your own search engine is hard (2004)

There are a lot of crawlers out there, and many of them are ill-behaved. When GoogleBot crawls your site, you get more visitors. When FizzBuzzBot/0.1.3 comes along, you’re more likely to get an overloaded server, weird URLs in your crash logs, spam, or any other manner of mess.

Small search engines getting blocked is just collateral damage from websites trying to solve this problem with a blunt ban-hammer.

bryanrasmussen on July 23, 2022 [–]

This is probably because, as I have learned here on HN, 1 in 4 programmers can't write a decent FizzBuzzBot and should never be hired.