why would it matter at all, given a crawler hits many pages on the same hostname?
the fact the initial request takes 250ms more is meaningless.
when you also take into effect that crawlers are either: bound on sleep() if they're friendly, bound on cpu if doing processing and you have a lot of money for bandwidth, or if you don't have much money, bound on bandwidth.
and given that crawlers tend to be massively parallelised, the DNS query could take minutes and you really still wouldn't care...
when you also take into effect that crawlers are either: bound on sleep() if they're friendly, bound on cpu if doing processing and you have a lot of money for bandwidth, or if you don't have much money, bound on bandwidth.
and given that crawlers tend to be massively parallelised, the DNS query could take minutes and you really still wouldn't care...
(go and read up on Amdahl's law)