Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
mc32
on Nov 19, 2018
|
parent
|
context
|
favorite
| on:
Web scraping case fails under Supreme Court's Dast...
There are a few forums where I don’t expect nor would prefer indexing. They may be niche forums, etc. Forums like reddit, of course I expect to be indexed. So, basically, it depends.
jimktrains2
on Nov 19, 2018
[–]
Isn't this what robots.txt was created to convey?
mc32
on Nov 19, 2018
|
parent
[–]
Yes. But not all scrapers and indexers respect that? So it’s still a problem.
baroffoos
on Nov 20, 2018
|
root
|
parent
[–]
Require a login/invite then
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: