Could they not run webhooks where my site can call to them when to do a full or ...

ceejayoz · on Oct 18, 2013

Sitemaps (http://www.sitemaps.org/) help a bit in that regard, as crawlers can check the sitemap and only crawl updated content.

unhappyhippie · on Oct 18, 2013

Thanks.

sergiosgc · on Oct 19, 2013

PUsH: http://en.wikipedia.org/wiki/PubSubHubbub

Google is the only crawler I see using the protocol on my site. It does make Google updates occur within minutes, so that alone is reason to implement push.

leephillips · on Oct 19, 2013

I thought this was only being used for RSS - interesting!