Crawlers ignoring robots.txt is abusive. That they then start scanning all docs ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		michael1999 4 months ago \| parent \| context \| favorite \| on: AI scrapers request commented scripts Crawlers ignoring robots.txt is abusive. That they then start scanning all docs for commented urls just adds to the pile of scummy behaviour.

tveyben 4 months ago [–]

Human behavior is interesting - me, me, me…

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact