We're kinda tied into DNSimple since we use an ALIAS record for our bare/naked/root domain. Amazon's Route53 supports aliases, but via a 301 redirect, which doesn't work in an SSL context (without browser warnings).
Nonetheless, we just spun up a Route53 zone, exported our zone from DNSimple, imported to Route53, and hand-migrated our ALIAS records to static A records in the new zone.
Not perfect or permanent, but we've gotten around the outage. Also, I just learned that pointhq has (seemingly-undocumented) support for ALIAS records in the same style as DNSimple, so this could be another avenue to explore.
Though they doesn't seem as innovative and nice as DNSimple. Really hope things work out for DNSimple (really like the idea of their beta feature GitHub sync).
How does their beta-feature with github work? I can't find any obvious link and it seems frustratingly close to something I offer over at https://dns-api.com/ ..
(I wrap Amazon's route53 with DNS entries read from github/gitbucket/similar.)
AWS Route 53 has added vanity DNS in the last few weeks, so you can make your DNS servers appear to be ns0.yourdomain.com rather than ns153.awsdomain.com (or whatever)
However, making your DNS servers to be responsible for serving their own DNS is a bit of an extra complexity and risk that no customer will ever care about..
"You can create generic "white label" name servers such as ns1.example.com and ns2.example.com, use them in your delegation set, and point them to your actual Route 53 name servers."
When you upload a new zone it will be assigned a set of nameservers - four. You can't choose what they are in advance, and you can't specify the TLDs. So you might end up with "ns-1933.awsdns-49.co.uk.", "ns-1109.awsdns-10.org.", or similar.
Does that answer the question? I'm a little hazy on what you're actually asking.
Old DnsMadeEasy customer here (used them for several years before moving to AWS). They were rock solid. We used them for hosting DNS for major cruise lines (fancy, very fancy ones), as well as other large Fortune 500 clients.
Their interface is pretty bad, but the backend is hardy.
Oof, really good point – I suppose I got lucky in that I was at dnsimple.com just a few hours ago for other purposes. Their DNS was still cached for me, and it all worked flawlessly. I didn't even suspect their outage extended to their web servers, though that makes perfect sense.
You can force your hosts file locally. Set "dnsimple.com" to "50.31.213.210". On Mac use "sudo pico /etc/hosts" and follow the formatting of the other entries.
As far as I understand, ipviking simply hosts honeypots around the world and uses those to graph "attacks" against IP blocks, etc.
I would very much like someone to correct me if this assumption is incorrect, because it'd be neat to actually watch targeted DDoS attacks, but I don't think that's what ipviking is offering.
Free solution that worked for me: Set up a free account on cloudflare.com, duplicate all dns records (thankfully I have a simple setup)... but next time I will keep a backup zone file!
FYI - Instead of an Alias record on DNSimple, CloudFlare will allow a CNAME record for the root domain using "CNAME flattening".
You can now set CloudFlare's DNS service to "bypass Cloudfare" on all records by clicking the icon so you don't get any of their magic (unless you want it).
Then add CloudFlare's 2 nameserves to your domain as your first 2 name servers. No need to remove dnsimple's name servers.
Now you have 2 DNS providers in case one fails, just make sure the records are the same across them both!
Hey thanks for the post, but how do I add CloudFlare's nameservers? DNSimple won't let me add NS records on the root domain: "You may only delegate subdomains". So in theory, I can fix the www subdomain, but my naked redirect to www won't happen if DNSimple is still down.
You would set your name servers with your registrar, such as godaddy. If dnsimple is your registrar as well as your DNS provider you may be out of luck until they come back online. The help page is here though: http://support.dnsimple.com/articles/setting-name-servers/
they are essentially the same thing -- both are "proprietary" names for the same feature, which is behind-the-scenes recursive CNAME chain lookups by the authoritative nameserver, to return A records directly.
DNS is so straightforward, so easily distributed, and so fundamental, that I'm always astounded when it's a single point of failure for so many operations.
I wonder how many of the affected companies do have redundant appservers and load balancers, but missed this piece of the puzzle...
dns is less easily distributed when fancy features like ALIAS (which dnsimple is widely known for) are in the mix. and wide distribution isn't enough to win vs truly volumetric attacks. it takes a lot of ports and compute to absorb 100Gbps+ attacks which are not uncommon against major providers.
DNSimple is widely know for the ALIAS pseudo-"record" because they invented it[1].
Small wonder that a proprietary syntactical sugar leaves you at the mercy of select vendors?
As for volumetric attacks: your point is correct, but is irrelevant if you're using multiple vendors, and a specific, single vendor is the target, like it appears here. Your other authoritative servers would be unaffected.
good luck finding any major online property or infrastructure that isn't making use of some kind of proprietary syntactical dns sugar. it doesn't mean you can't span providers, but it does mean it takes a lot more work to do so.
anyway, you're not wrong, the best approach to mitigate this kind of thing is to leverage multiple dns networks. but doing so is not easy unless the application is still using dns like it was in 1995, and that is increasingly rarely the case.
Using a WWW subdomain with CNAMEs accomplishes effectively the same thing as using ALIAS on an apex domain name, and doesn't rely on anything out-of-spec or proprietary, making it easier to serve redundantly. (Did you ever wonder why google.com and facebook.com redirect to www?)
(Or is there more to ALIAS than that, which wasn't on the page in GP? Happy to be corrected if so)
you're correct about ALIAS (although practically, it doesn't matter: people are going to use the apex whether it's proper or not at this point). i'm more referring to other complex usually-proprietary capabilities of big dns providers, especially traffic routing features. routing semantics are generally not translatable across providers, and if you're using dns based routing (as most cdns, major web properties, etc are) then doing multi-network dns gets a lot harder. if you're amazon, you write and maintain a bunch of code to span providers. if you're not, the barrier to multi-network is high if you're doing more than static dns.
You're right, but people want to get fancy with hosting at the apex (domain.com), even though it kills important functionality (CNAMES) forcing the adoption of hacks (ALIAS and ANAME records).
I'm surprised more places don't run their own DNS. It's not that difficult to do and it means you don't have to rely on another third party for service.
Well if this is an attack to get at one of DNSSimple's customers, running your own DNS would be a much easier target. Which is to say that if you were the target, you would already be hard dead by now rather than struggling as DNSSimple deploys defenses.
I agree though that it is a pretty simple service to run for a small domain.
If you're being targetted directly, then all of your services need to be DDOS proof, not just DNS. The more third parties you add, the more likely you are to be taken out by accident. If you have your own web server, you should dump Bind or PowerDNS on it and write a zone file. Problem solved.
Not that difficult for whom? Great if you have the in-house resources to devote to managing your own DNS (and can't put them to better use elsewhere), but that is not the case for the vast majority of us. The fact that so many of us use PaaS companies like Heroku should be a pretty big indicator that most platform-related engineering is not going to happen in-house under a certain scale.
If you can figure out DNS on any hosting service's page, you can do it in multiple places.
You could pay GoDaddy, Amazon Route 53, and DNSSimple to all host your records, for example... Management would be slower and manual, but people without resources for "managing your own DNS" won't be changing records that frequently anyways.
The odds of all three going down at once should help your uptime, yes?
Not that difficult for somebody who is a sysadmin or developer. I would hope that services like Heroku offer DNS too, but I wouldn't know. If your site is already hosted at Heroku, it would be better for them to deal with your DNS so that you don't introduce additional third parties. That is my point.
I moved from Zerigo to DNSimple, and it's been awesome until now!
What can you do to prevent this in future? Can you run multiple DNS providers simultaneously? So, ns1/ns2 go to DNSimple, and ns3/ns4 go to another provider?
I replicated our DNS on DNSMadeEasy, which I've used in the past on a very high-traffic website. They have an incredible track record for uptime. And they're really cheap.
We manually exported our DNS configuration from DNSimple as a single file and imported it to DNSMadeEasy. Pretty easy transition, although we have to make future changes in two places.
I set 2 nameservers on the old host and 2 on the new, and am keeping that configuration for automatic failover in the future. The DNS propagated fairly quickly and our site was up and running for many users within an hour. Glad that's over with.
DNSimple does not currently support zone transfers (see http://support.dnsimple.com/articles/master-slave-support/), but you should write an e-mail to the support (address is on the linked page) requesting it :) With dozens of domains, managing multiple DNS servers is a pain. Zone transfers make this very easy.
Publishing an incomplete copy of your zone seems like a bad idea, an ANY dig at the zone apex is going to retrieve less than 30% of the records (on average) for real-world zones. Negative responses in DNS are cacheable - so you can easily poison yourself.
If you really really want to do it anyway, most caches use either the TTL on your SOA record, or the final field in the SOA record as the negative cache TTL; so lower both of those values to something like 60 seconds.
Hmm, never considered that. Is it a slow, static process for ISPs to do that? I just assumed that they ran more-or-less stock DNS resolvers with in-memory caches.
They're in ServerCentral's datacenter and ServerCentral is very much multi homed. They wouldn't gain anything by doing native BGP to all these peers in the exact same datacenter when SC's backbone will handle this stuff for them.
Doing native BGP would allow them to anycast, which would increase their reliability and allow them to sink traffic much more easily. DDOS traffic sink starts announcing your AS and anycast IP block close to the traffic source, sinking that traffic and allowing real traffic through.
I'm pretty sure they're only in one of Server Central's datacenters. Anycast won't help. That's why I said they don't gain anything by directly peering.
Does anyone have a simple explanation or link to an article / blog that explains the naked domain / ALIAS "problem" that DNSSimple solves? I recently set up DNS with DNSimple (due to nudging by Heroku) and am affected by this DDoS. I am still struggling to understand the exact nature of this issue. All of Heroku's documentation is pretty cryptic (to me):
"Some DNS hosts provide a way to get CNAME-like functionality at the zone apex using a custom record type. " .. and then on to suggest DNSimple as their first suggestion.
http://blog.cloudflare.com/introducing-cname-flattening-rfc-... is a reasonable explanation. fundamentally a CNAME says "when you get queries for this name, go look at this other name instead". among other things, doing a CNAME at the zone apex means resolvers can't then find your NS, MX, or other records at the apex, which is problematic.
For those wondering about alternatives to ALIAS: if you use a www subdomain, then you can simply use CNAMEs. (Though the appearance is a matter of taste...)
Yes, but if someone visits your apex domain directly, you want to redirect him/her to the www version... And if you use cloud based hosting where the IP can change quite often, that's a pain to maintain manually.
Anyone switching from DNSimple? I really don't want to, but we've been down for almost 3 hours. I've seen chatter about Cloudfare and it looks pretty good, reviews?
We switched periscope.io from DNSimple to Amazon Route 53. DNSimple doesn't have an exporter so it took about an hour, including having one engineer review the other engineer's work.
Many customers were able to resolve the domain in the minutes immediately following the switch, and the rest seem to be trickling in.
We switched baremetrics.io over to AWS/Route53 as well. It didn't take long (NS switch might still be propagating) but not having ALIAS support is a bit of a drag.
DNSimple is my registrar and (was my only) DNS provider. Now that they're back up I've exported the zone file and imported it to route 53 for redundancy in case this happens again. I also I updated the name servers in DNSimple to be 2 route 53, and 2 DNSimple, in that order. Is that the right way to do it? Does the order of the NS records matter? I set them up so that they're in the same order in both places.
That should be right. I'm doing the same thing with CloudFlare. However, it appears that DNSimple won't keep your secondary name servers in the order entered. They are showing for me sorted alphabetically. Apparently, they are working on providing this (http://blog.dnsimple.com/2014/12/incident-report-ddos/) for failover in case of an event like this again.
I wrote a follow-up article about what we at Canopy.co learned from this incident. Check it out (this covers and expands on some of the ideas talked about here):
Unfortunately, it's not the first time it happens, my app is down and customers unhappy.
I always wonder, why is it that someone wants to attack a small company like DNSimple ? Is it that they were blackmailed and did not surrender to the criminals? If so, why would anyone be interested in blackmailing such a small company?
More likely, I'd guess based on past experience, it's that someone wanted to take down one of their customers, and decided (or found) that the weakest link was the DNS provider.
It could also be an attack against one of their customers, again for the same reasons. DNS is a very weak point of the overall system so a very good place to target.
My site is currently offline from this attack. I am considering that providers like DNSimple simply cannot provide the networking availability to mitigate these kinds of attacks. This is because the solution to these kinds of attacks, often, is that you need a larger pipe than the attackers. Very few people are good at both <some service> and having a huge pipe. For web apps, you can use CloudFlare, which does have a bigger pipe and is designed to mitigate this. But DNS is not a web app, so you can't just put DNSimple behind CloudFlare.
Hence, I'm going to try CloudFlare (assuming they take over DNS hosting, I need to check) and Google Cloud DNS, because then all parts of my site (from DNS to CSS hosting) will be with providers with bigger pipes than attackers can create. Hopefully that will prevent this kind of attack from taking my site down.
You can use two or more DNS providers, but if you do try not to list more than two name servers from any one provider when you delegate. You should also take care to do the same in your own zones "NS" record set. Some providers don't let you change the NS record set, so avoid those providers.
The reason for this is that resolvers will generally try at least three different name servers before giving up, so if you have three or more from a single provider that may not help.
There is also a big caveat to consider: once you use two more providers, whenever you need to make rapid DNS changes for your own availability reasons you will need to wait for the slowest-to-update provider.
have multiple providers? VPSs aren't expensive and setting up bind9 or bind10 or whatever authoritative name serving daemon isn't rocket science (well, you CAN do it wrong).
my experience shows, that at least 3 servers with 3 different providers is good enough. and "providers" I mean different company, city, datacenter, transit provider...
trusting single entity with anything (even if they say that they have many servers blabla... geologically blabla...)... well, you have situation like this right now :)
"DDoS attacks...will generally fall into one of three broad categories:
Volumetric Attacks: Attempt to consume the bandwidth either within the target network/service, or between the target network/service and the rest of the Internet. These attacks are simply about causing congestion."
One of the most valuable things I learned in my career was to never kick your competitors when they are down. It upsets the karma gods, and makes you look like a total douchebag.
The best thing you can do right now is to reach out and offer your help, privately. Even from a selfish perspective, you'll learn a lot about the attack that is taking them down now which will help you out when the targeted customer inevitably signs up for your service.
EDIT: The parent comment was spam from Kris Beevers at NSone.
hi. i did not mean to spam or kick dnsimple, we know them and they are a great company and service. we are actively receiving inbound queries about this from folks asking for help, so thought it made sense to chime in publicly here. but you're right, i should have kept it on topic to the discussion at hand instead of offering anything else up.
you're not wrong: in this industry you never kick your competitors when they're down, everyone is subject to the same constraints, attacks, and complications. that wasn't my intention and i said so in the post.
That seems a little extreme; perhaps what he did was in poor taste, but to use karma and account age as an barometer for your business decisions seems crazy (or an arbiter in an internet catfight).
Nonetheless, we just spun up a Route53 zone, exported our zone from DNSimple, imported to Route53, and hand-migrated our ALIAS records to static A records in the new zone.
Not perfect or permanent, but we've gotten around the outage. Also, I just learned that pointhq has (seemingly-undocumented) support for ALIAS records in the same style as DNSimple, so this could be another avenue to explore.