Hacker News new | past | comments | ask | show | jobs | submit login

The idea that a robots.txt will save you is laughable.



Agreed. At best, you can disallow: / and hope they're polite enough to listen.

I can't seem to find anything on OpenAI's crawler agent, so I'm skeptical they're considering robots.txt at all.


Even if they abide, this is capitalism. Somebody who wants an edge won't. Or OpenAI or Google will get desperate and stop abiding.


True. Robots.txt is already a very weak thing. I disallow all access using robots.txt, but there are many crawlers who ignore it and I have to maintain an overt blocklist for them.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: