Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It's funny how scraping can be illegal yet Google and Microsoft and Yahoo and DuckDuckGo, and every other search engine have a monopoly on it. It's legal for them, because .....? Money money money! Google even hotlinks images for Google images... pretty hilarious stuff.


Maybe scrapers should respect robots.txt and go away if there's a `scraperbot` entry, the same way Google goes away if you put `googlebot` in there.


You can block them in your robot.txt if you desire... it’s not the same thing, websites want to be indexed




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: