It's funny how scraping can be illegal yet Google and Microsoft and Yahoo and Du... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		goldenkey on July 11, 2020 \| parent \| context \| favorite \| on: Scraping Isn't Hacking It's funny how scraping can be illegal yet Google and Microsoft and Yahoo and DuckDuckGo, and every other search engine have a monopoly on it. It's legal for them, because .....? Money money money! Google even hotlinks images for Google images... pretty hilarious stuff.

dafoex on July 12, 2020 | [–]

Maybe scrapers should respect robots.txt and go away if there's a `scraperbot` entry, the same way Google goes away if you put `googlebot` in there.

Google234 on July 12, 2020 | [–]

You can block them in your robot.txt if you desire... it’s not the same thing, websites want to be indexed

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact