We had a 500 cluster web crawler using us. Probably doing about a billion queries a day. They had the option to pay us or switch to Google Public DNS. Guess which they did? :-)
A few hundred, even if a few hundred per minute, is not a problem.
I have operated a rather large crawl cluster on EC2 (several hundred nodes), and I don't see why anyone would point one at a third party DNS service.
we operated for about 6 months using the default resolvers than come with the AMIs, then Amazon contacted us, telling us they wanted us to stop using their recursive nameservers... apt-get install bind, point at 127.0.0.1, redeploy AMI, done (in under 5 minutes).
it isn't as if a crawler cares about an extra 250ms from non-cached entries, and the bandwidth from DNS is trivial* compared to that of downloading pages.
... so why would anyone ever offer to pay you/anyone else for a recursive DNS service? it's a trivial problem...
(* say 32 bytes for the query, 64 bytes for the response... 1B lookups is ~64gb, or $3.50 with Amazon's very expensive bandwidth costs)
why would it matter at all, given a crawler hits many pages on the same hostname?
the fact the initial request takes 250ms more is meaningless.
when you also take into effect that crawlers are either: bound on sleep() if they're friendly, bound on cpu if doing processing and you have a lot of money for bandwidth, or if you don't have much money, bound on bandwidth.
and given that crawlers tend to be massively parallelised, the DNS query could take minutes and you really still wouldn't care...
Inbound bandwidth to Amazon has been free for a while. Anywhere but Asia and South America, your posited 32 byte query adds up to about $0.36 per billion at their most expensive $0.12/GB tier.
No, he's talking about a different DNS - OpenDNS. It's a confusing thread - the top comment is by a guy running OpenDNS. OpenDNS is a DNS service which resolves typos, blocks typo fishers, and sends empty pages to an OpenDNS ad page. They also have a bunch of options and controls, letting you block some sites (useful for employers, schools, and parents).
It sounds kind of sleazy (redirecting "no record" to an ad seems a little off to me), but the guys running it are apparently not. The prejudice against redirects to ad pages is more a result of ISPs who take your money and still give you ads, unlike OpenDNS which is free.
Being in the information security world, I have to wonder what percentage of these requests are based in malware? I know at least the latest version of ZeroAccess/Max++/Sirefef (which we managed to get before the AV vendors released definitions for it) uses it quite heavily. That's one of the symptoms we used to diagnose computers from a strictly network-level standpoint. No one on our network should be using Google DNS, so any computers who were making requests to 8.8.8.8 were likely infected (confirmed using other signatures).
That amounted to about 100 requests every day per infected computer just from us, and ZeroAccess isn't the only one doing it (and isn't a rare trojan).
I just switched to Google DNS. Before I switched I got a 38ms response time when I ping google.com, now I get 285ms? I'm in the UK. Is this why it's slow?
It's possible that you are suffering from mistaken redirection to a different CDN.
You should read up on how CDNs work (http://en.wikipedia.org/wiki/Content_delivery_network) but the gist is essentially that your DNS server determines which content node you're routed to. If your DNS server is your ISP's DNS server it's very likely that you're geographically close.
However, public DNS is usually anycasted to more general regions. In this case, you may be then routed to a content node which is close to the DNS node, but farther from you.
Google actually does support the EDNS extension in order to help solve this problem, but I find it unlikely that ping supports the extension.
This is a big problem with public DNS services. My research group will be releasing a project soon which seeks to alleviate the problem and dynamically choose DNS based on what provides you the best performance.
Edit: Actually I think I may be wrong about one point: IIRC the actual DNS clients don't need to support EDNS-client-subnet in order for it to be used. Only the DNS and authoritative DNS/CDN need to support it. Usually that's a problem because most of the major CDNs don't support it yet, but Google actually does support it on both their CDN and public DNS. Therefore you should be routed based on your prefix when talking to google.com, not the anycasted 8.8.8.8 node. Thus, I have no idea what's going wrong.
DNS has nothing to do with ping response times. A ping requests the IP address from the DNS server and then does the ping. On most systems the DNS is cached locally, so multiple requests will use the same information.
I don't think that DNS lookup time is included in ping's latency numbers. It probably does the lookup once per run, and then caches the IP. It wouldn't make sense to to the DNS lookup once per ping.
I would be interested if statistical data gleaned from DNS makes it's way into any other service areas. DNS would seem like a useful way to rank the popularity of web sites, I am sure there are some interesting enhancements that could be made using that data.
> We don't correlate or combine your information from the temporary or permanent logs with any other data that Google might have about your use of other services, such as data from Web Search and data from advertising on the Google content network.
And they say that the logs are only used for debugging, DoS protection and abuse.
I did not mean "my queries" being "shared" but the query collection/archive as a whole. It would be another source to know what domains the people visit.
> And they say that the logs are only used for debugging, DoS protection and abuse.
I for one am very thankful for this DNS. Ever since moving to China, 8.8.8.8 has useful for getting around various flaws of the internet experience here. Even with a paid VPN, it's nice to have an always working DNS server.
Obviously it cannot be faster than a local DNS cache/server (for the RTT). What can be significantly faster is forwarding queries to Google Public DNS instead of your ISP. At least in my experience, ISPs often have slow and overloaded servers which are not well maintained. Recursively resolving completely on your own is often even slower than either of those two choices.
I have been running unbound for a month, and there is no noticeable difference in resolving time. This article provides a good read: http://linuxmafia.com/~rick/googledns.html
45ms beats the hell out of the hundreds, or thousands, that crappy Irish ISPs deliver from their own 'local' name servers. When they're working at all.
especially one with such uniformly poor latency when compared to ISP-cobbled together crap it's supposed to replace, anywhere I've ever seen it deployed.
That seems overly negative: Google DNS is consistently better performing for me than other public DNS services except for Level 3's 4.2.2.1 &c because I'm on their network.
Google's public DNS uses anycast, so the performance should be good in most places, perhaps your experience is the result of your geographical location or ISP's network?
> other public DNS services except for Level 3's 4.2.2.1
Level 3 doesn't actually run a public DNS service, they run DNS servers that happen to permit requests from non-customers. You'd be well advised not to use it outside L3's network.
What do you mean by poisoned? For me it's the contrary, I switch to Google Public DNS whenever I'm using an ISP with "lying" DNS (returning a search page instead of NXDOMAIN).
The basic technique takes advantage of the fact that DNS allows you to provide additional information in a response so the response for ev1l.hax0rs.org can return a reply which says "This is handled by ns.reddit.com. Oh, by the way, ns.reddit.com is 1.2.3.4"; any server which doesn't properly validate that last part would add the incorrect ns.reddit.com record to its local cache and potentially use it to handle requests for other clients.
That's a better question for the original poster - I described the generic technique but haven't heard of anyone successfully applying it to Google Public DNS.
Our growth: http://i.imgur.com/znfu9.png