Hacker News new | past | comments | ask | show | jobs | submit login 

    User-agent: Mediapartners-Google
    Disallow: 

    User-agent: *
    Disallow: /search
    Disallow: /

    User-Agent: googlebot
    Disallow: /search
    Allow: /
Woah, that is surprising. I note Bing has blogspot in its index anyway. Perhaps they use the ATOM API when they see a Blogspot URL? (technically not 'crawling')



Where are you getting that? It doesn't match what I'm seeing. http://googleblog.blogspot.com/robots.txt


Googled for "blogspot", picked first random domain I saw, "weliveyoung.blogspot.com", fetched "weliveyoung.blogspot.com/robots.txt" with curl, got a redirect to "weliveyoung.blogspot.co.uk/robots.txt", fetched that, voila.

Perhaps there is a user setting that controls it.


Looks like you can use whatever you want. The one I linked to is the default.

https://support.google.com/blogger/bin/answer.py?hl=en&a...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: