If everybody does this then a new search engine will never come to be. You better hope Google is the best search engine anyone could have ever created.
As a search engine developer that has tried to compete with Google in the past, I find this disheartening.
I agree with blocking anyone doing bad. But you are cutting out the good with the bad.
Can you tell us a little bit about your project to compete with Google if it's ok for you ? "try to compete with Google" is exactly what I'm doing right now and I'm just curious.
You were not able to crawl hundreds of thousands of websites because your crawler was disallowed by their robots.txt but major crawlers were allowed to do so ?
Blocking is never a sustainable strategy but sometimes you have to kill some good cells with the cancerous ones. Chalk it up to collateral damage. It's the same situation with forum/comment spam often the IP falls within a range used by OVH so when it gets out of hand I block the entire IP block.
As a search engine developer that has tried to compete with Google in the past, I find this disheartening.
I agree with blocking anyone doing bad. But you are cutting out the good with the bad.