Hacker News new | past | comments | ask | show | jobs | submit login

Robots stops a page from being crawled the noindex tag stops it getting into the index.

Google is also slow to honour 404 and drop pages which can hang around for ages, Bing is much faster to remove 404 pages.




Usually, creating "410 Gone"[0] response for the URL and running the URL through the URL Inspection Tool [1] can help make things a bit faster. But yeah, it does take a while to get these 404s removed.

[0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410 [1] https://support.google.com/webmasters/answer/6065812?hl=en


That distinction exists in many systems. E.g. for cloud events, 404 is considered with skepticism because it could be a race condition in provisioning or transient issue whereas 410 requires data streams to be cut off.


Then you should serve 503


5xx means that the server made a mistake. 4xx means that the caller made a mistake. Sending a request to a GONE url is canonically classified as a “user” or sender error.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: