Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt

Is this a common mistake?



Yes, from the server side's mishandling of TXT extension. Probably the server put the MIME type in the HTTP header as "HTML" instead of TXT, and the browser renders the page as such.


From the headers:

    Content-Type: text/html;charset=UTF-8
It also tries to set no less than four cookies.


It just issues a "HTTP/1.1 302 Moved Temporarily" directed to their homepage. Requesting an invalid file such as "robots.txtsdfa32r523" has the same effect, so they probably don't have a robots file at all.



Huh? No, it is a regular robots.txt file


It redirects requests from the UK.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: