I'm interning at a major managed DNS provider and this actually happens more often than you think. Most of the time it ends by working with telcos like level3, cogent etc, blackholing routes, sacrificing a certain geographical location in favor of keeping others alive and basically just waiting it out. It's interesting how because of the nature of the internet, DDoSes are hard to deal with.
From 2001-2003 I led development on Arbor Peakflow DoS, which (when I left, at least) was far and away the most popular tool tier-1 ISPs used to manage these kinds of attacks, and my the experience I can relate is:
* These attacks happen with shocking frequency (the major incentive ISPs have to mitigate them is not angry customers, but rather the drag it creates on their third-tier engineering staff to hunt down DDoS sources and craft ACLs for them)
* There is actually not a whole hell of a lot you can do about them, even if you light up a whole tier 1 core network with mitigation tools. At the end of the day, a well-crafted DDoS attack looks pretty much identical to normal traffic; if you can black-hole China to defeat an attack, it wasn't very cluefully done.
Regarding your statement that it wasn't very cluefully done, isn't that sort of a hit and miss area? You can't really control much where your malware spreads (and thus where your bots are).
As a sidenote, I've been reading your posts for a while now and it seems you always have something insightful to say - cheers!
it's not difficult to control where malware spreads. directly attacking specific ranges is one easy method, as is coding the limitation into your bot or only sending commands to the ones in a certain range.
There's an anti-DDOS service out there, Progent or something. Part of their solution adds all of APNIC's ip addresses into your router's bogon list when you're under attack. It turns out for most DDOS's if you turn off china, the attack just ends.
I think you're thinking of Prolexic, and my recommendation is that you take any of their claims with a heaping helping of salt. Things may have changed in the 5 years since I've totally forgotten everything I knew about DoS mitigation, but it wasn't the case then that you could simply black hole China to evade them.
A more common mechanism used by real networks to mitigate these attacks in production is to pinpoint the target of the attack and offramp its traffic in the ISP core to a "regional scrubbing center" where advanced filtering tools (for instance, packet filters that can handle ACLs with high tens of thousands of terms) can try to sort through the crap.
At my colo that is exactly the sort of strategy they use, they have bought a bunch of routers that use FPGA based filters for that.
The one scenario I can think of where that might be a problem is if they'd start flooding all the known hosts in a network for the specific purpose of overwhelming the routers. And even those hardware based filtering tools have upper limits.
There's really no need to do that. A large enough BotNet can take out almost any host doing nothing more than connecting and doing a GET / HTTP/1.1. Few websites can handle a sudden surge in traffic from 100,000 bots.
The idea behind an ACL for such an attack is that the same hosts will be used over and over again, taken from a large pool of zombies. So, let's take your example of 100,000 bots, an ACL in the upstream router (as seen from the host) could be used to checked against to identify those packets from the zombies and discard them. After all, a single attack of all 100,000 bots at once will just bring the host to its knees for the time-out of the connections and then it will bounce back up again. So, to increase the effectiveness of the attack they reconnect after every lost connection asking for another resource. If they're smart they'll vary agent strings and other characteristics to make it hard to narrow down who is and who is not legit.
Initially you don't have much to go on during such an attack and packet filtering is a reasonably expensive operation when you want to do it for a large number of hosts. So the strategy is to route all the traffic destined for that particular host through a router that has ACLs that are large enough to hold the total IP list for the botnet that is attacking the host, as these IPs become identified.
You don't want to route all your traffic through there because then you'd have to do the relatively expensive filtering on all of the packets, even those not destined for that particular host.
Now if an attacker were targeting the hosting facility they could thwart this strategy by sending requests to a larger number of hosts in the network in order to make life much harder for the crew fighting the attack. After all, you can't partition the problem anymore in to a portion that is targeted to the host and 'normal' traffic, effectively all the traffic could be bot traffic or it could be normal traffic, for all the receiving hosts.
To be able to partition the problem into a smaller one where you can let say 90% or more of the traffic through unfiltered and only concentrate on the remaining 10% would make solving it a bit easier. On the other hand if the attackers are silly enough to re-use the same bots to attack different hosts they've actually given you a clue as to which IPs are bots.
I'm wondering what the implications of DDOS are for website owners. What if you're on EC2 for example, will you be charged for the 300 TB of traffic? If so that would be an easy way to bankrupt a startup.
The implications for website owners are that their web sites were temporary unavailable.
Traffic simply does not reach the web site. Even normal traffic.
For example, you type in your browser www.mywebsite.com.
DNSMadeEasy would normally resolve it to your IP address (e.g. 111.222.123.12). But because of the DDoS attach -- mywebsite.com cannot be resolved into any IP address and you cannot open www.mywebsite.com at all.
That's the implication is there is a DDoS attack on your DNS provider, but I think FooBarWidget was inquiring about the implications of a DDoS attack on your website, and specifically if you would be charged for the bandwidth consumed by the attack.
You can - but not everywhere - negotiate that the service you pay for includes protection against DDOS attacks and that it's up to your provider to protect you. You'll pay a larger fee per mbit because they'll need to do more work for you in case you get hit but it might be worth it.
My policy for this is to simply lock the doors and hide for an hour if my network traffic averages over 2 Mbps for 5 minutes. I would also send myself an email. So far this has never happened.
get akamized. you can also build traffic limits into your web stack or OS or network gear if you're afraid of traffic ramps hurting your wallet. like tptacek said, a good DDOS looks like normal traffic, so this could happen if you got slashdotted by 10 different news sites.
No wonder most of my sites were down or not functioning very well this morning. Hope they get everything operational soon, DNSMadeEasy is a great provider.
Can someone please explain to me what DNSMadeEasy actually does?
Is a domain provider like 1and1 a client of theirs or is my hosting provider a client?
This entry on Hacker News led me to wikipedia where I went from an article about Level3 to an article about Tier 1 Network to Internet Backbone. I feel like I'm on the verge of understanding more about how the internet works but it's all a bit above my head.
They're an anycast dns provider- basically the only kind of third-party DNS service that's worth anything at all. They have some neat domains-related tools that hook into their infrastructure that's kindof a pain to do yourself (even if you've got your own anycast-capable network)
Probably some motive against one of the customers. If you can extort some cash out of a gambling site with short dns-ttl it could be worth your effort.
Maybe the target's data center is more ddos-proof than the one from easydns, extortionists go for the weakest link.
Back in the olden days of IRC, there was times when entire regions of the internet were taken offline resulting from a ddos for personal vendettas. For exactly that reason too - they attack the weakest link and often it isn't the host directly.
In this case - it really could be anything. The cost of one of these attacks is next to zero. Rarely will the botnet owner lose any machines resulting from an attack. The unfortunate thing after one of these attacks is you have no way of preventing it or going after the source.
it's fairly large. There are really two factors in a DoS, though, total throughput and packet size. Obviously, your incoming pipe can only handle a certain total throughput, I mean, that's what most of us get billed on.
However, most routers and firewalls also have a limit on packets per second they can process, on top of the throughput limits. I've got a 100Mbps commit on a 1000mbps pipe, and I can handle 1000Mbps of 'normal' traffic... but I got taken out a month back by a 200Mbps DDos that used very small packets. My router couldn't handle it. (now, if I had spent money on a better router, it wouldn't be a problem. As far as I can tell, even, a reasonable software router could have handled it.)
Another way to measure this is the capacity required to absorb the attack. You can get he.net bandwidth for around a thousand dollars a month per gigabit, and he.net is about as cheap as bandwidth gets, so to soak a 50 gigabit attack, you'd have to have fifty thousand dollars a month of spare capacity.
(I'm sure there are further discounts available between the 1GiB and the 50GiB tier... but you get the idea. )
Considering the fact I've seen this attack first hand. I can tell you a couple of things about it's strength.
It's very flexible one minute they are sending packet size 1500 bytes udp, the other they are sending 48 bytes syn tcp 80. However, filtering them with a Firewall is not hard at all since they do have packet patterns you can detect, but even if you can find a firewall and have it on the edge of your network traffic is STILL reaching your network and if you can handle 50Gbps of traffic all coming from a couple of different ASn than "wow".
50gbps is pretty massive. Clearly somebody did something horrifically insulting to China like mention that they could maybe possible consider treating Tibet a little bit more humanely and stop turning it into one big whorehouse.. make a statement like that, and all of china ddos's you.
No. (having lived in China for multiple years) Pretty much all the political issues the West thinks of WRT China are not even given a cursory thought by most Chinese. China just isn't very political. Taiwan is basically a non-issue. Tibet is completely a non-issue.
More to the topic: It's much more likely that the motivations are financial.
And who do you think pays these people to ddos sites like slideshare and posterous? (both of whom have been my customers during massive ddos's from China). I'm sure there's like one posting somewhere on Posterous that someone in the Chinese government didn't like, so they paid their team of script kiddies for time on their botnets to bully Posterous around. The same thing happened to Slideshare a few years ago.
It's about China bullying people around and trying to censor the internet.
Your assertion that the China gov sponsors and pays for this is mostly unwarranted. I say mostly, as I can see why someone without experience in what goes on internal to the China gov may think such things. But you still have no evidence. Most Chinese Windows XP installs are virus platforms ready made to be transformed into a bot-net army. These PCs could be controlled by just about anyone/anywhere. The main reasons they are virus ridden is (1) the PCs use unlicensed copies of Windows and MS does not allow updates and (2) Chinese software add-ons (browser tools, chat tools, etc) are particularly vulnerable, many times by design to allow easy access by the distributor, and (3) most software installed is pirated which may also contain virus payloads.
As an example of how outdated a typical Windows XP install is in China, I'm running a site which has 95% traffic from China. Over 62% of users are on IE6.
So then, patriotic hackers (riiiiiiiiiight) like to target American websites that are critical of the Chinese government because they feel a sense of nationalistic pride? Bullshit.
It's serious but not the end of the world. Of course if your uplink is smaller than that it might be a real problem. Then you'll have to talk to your upstream provider to do the filtering job for you.
The hosting facilities where I have my servers would class this one as 'just another days work'.
Sites that are routinely targeted for blackmail because they make lots of money have dealing with attacks like this down to a science. Of course they're not going to go out of their way to advertise that it happens all the time to protect their business interests, so that's why you may not have heard about it.
Banks and other financial institutions, gambling sites, large porn sites, top 100 websites and sites that are either vulnerable to brand damage or that have a lot of turnover see an awful lot of this.
50Gbps is at most 50K zombies or so, that's really not that bad.
The largest attacks against sites that I know of used a million ips and more. That's a wholly different kettle of fish and starts to be a real problem because even hardware based packet filtering (Thanks force10!) has its limits.
Additional notes (too late to edit the comment above):
After a call with one of my hosting providers (yes, on a Sunday at that, how is that for service), they saw the 40Gbit barrier broken somewhere at the end of 2007, today they're prepared for a multiple of that but he says that because they are that well prepared they've become less of a target.
They've invested a very large sum of money in infrastructural components specifically to deal with DDOS attacks at the hardware level, and though he doesn't rule out the possibility that they'll be one day facing one they can't deal with he doesn't seem overly worried, he does not want to claim any upper limit.
The countries they've seen the most trouble from are hard to pin down, but apparently the former USSR states and China are pretty high on hist list for the 'bot masters'.
Extortion seems to have arrived on the internet to stay, if you're a small player and you become successful you'd better be prepared, sooner or later you'll be a target.
Even smaller websites can easily get 2 to 10 Gbps ddos attacks aimed at them, the first time this happened to me I was pretty happy that all that happened was that I received an email from my ISP informing me of the fact without any loss of service.
I've done so in the past and I have already remarked somewhere earlier that I feel a bit uncomfortable about mentioning this business on HN because it feels a bit like advertising (and it's owned by a former employee and a bunch of his friends of mine so I'm not exactly impartial).
If you really want to know please drop me a line (email in my profile).
Not a single provider in the world can handle this kind of attack peacefully without service interruption except China Telecom and China Union.
Your calculations can't be more off about 50K bots, our counts showed more than 100K we couldn't count after that due to limitation in software.
My business has been dealing with DDOS attacks since 2002 we pretty much saw the brunt of every kind of new attack that came online. The only thing that can be compared with magnitude to this is the DNS Amplification attack, but that was limited in it's impact considering the source of the attack was diverse not from one geo area.
Force10 will sell you a router that will fend off a DDOS attack with over a million zombies. It's going to cost you though.
Yes, there will be an interruption, but you will be able to get the situation back under control while the attack is still in progress. You will need your upstreams/peers to collaborate.
If you're on the Cisco platform, then good luck to you.
Edit: interesting thread, apparently the reason it's 30gbps is that it's maxing out the link from China telecom, so legitimate Chinese customers are getting traffic dropped. FBI are involved. Suggested solution by some is to get traffic routed over more intercontinental links possibly via peering agreement with China Telecom, and beyond that political pressure.
I wonder if, long term, DDOS attacks will be to net neutrality as spam was to nice-and-easy e-mail setups. Now ISPs are running spam blacklists, blocking entire other ISPs at times, and it's a nightmare to run your own SMTP server and deliver mail reliably without jumping through hoops. If DDOS attacks become more and more annoying, they could be used as an excuse to violate net neutrality.
Really good spam e-mail is nearly indistinguishable from normal e-mail. Crazy layers of whitelisting/blacklisting/new DNS settings/laws/software and policies have been piled on top of the mail system to reduce the nefarious effects of spam sent en masse.
While spam and DDOS attacks aren't directly comparable in terms of what they are, I'm speculating in terms of what negative effects continued and escalating DDOS attacks could have again in terms of laws, policies, white/blacklisting of entire networks/countries, and so forth.
I wonder if they're going to change the text on their website:
"A DNS service with a 99.9999% uptime history is just the start! DNS Made Easy is so confident of it's uptime record that we offer the best service level agreement in the industry. That is why all businesses that require stable DNS decide to use DNS Made Easy. We have an industry leading 100% uptime gaurantee and will credit all accounts 500% of the downtime."
