Disclaimer: VideoLAN president and lead VLC developer here.
The attack started on our new mirroring system (powered by mirrorbrain) 2.5 days ago, during the night (after 2am, so we were sleeping).
We were woken up (OP and I) in the morning by many mirrors complaining of high bandwidth use. The actual number of requests was not that high (400 req/s), but the botnet was downloading the whole vlc.exe, aka 22MB. So, we were at around 70Gbps during the night, in average.
Afterwards, North America got up, and things got worse. We had up to 1660 req/s, so around 292 Gbps...
This is very weird for a DDoS, to be honest.
Our front machine that splits the down mirrors was taking most of the load, and we were able to find the patterns to drop the botnet connexions, in order to not kill too much our mirrors. I won't discuss too much of the patterns, as you can imagine, but as usual, I'll be happy to discuss it IRL or by mail.
Tweaking the front server was also important to reduce the number of open connexions, to not kill our server.
2.5 days after, the attack is still going on, with an average of 500req/s.
The video was done using logstalgia, using scripts of OP, on my machine (<troll>he was running eclipse, he couldn't do both at the same time :)</troll>).
Is there a reason VLC doesn't offer a Bittorrent download option, by the way? As far as I know VLC's audience is pretty technologically-inclined, so they would understand what a magnet link is, and you'd be able to just switch off the archive-download option for a while in cases like this. (As a bonus, if you embed a magnet link in your page, you're implicitly embedding a SHA-verification of the resulting torrent file.)
That's precisely what I don't get in this case, especially. The source is out there, there are other people mirroring vlc. So if you're trying to censor it, you're going to fail pretty hard. In the absence of that, what can it be?
I don't know what request throttling solutions are available for nginx. But, for apache there are modules and scripts/code that can rate limit traffic and requests from specific IPs. Depending on the size of the botnet attacking, rate limiting requests by IP or network may have been a way to mitigate the DDoS effects. Again, depending on the specifics, you should be able to use iptables (if you're running Linux) to limit requests per source IP or IP range. The type of actions you can take with iptables are really quite robust.
I've had to deploy request rate limiting solutions in the past because of well meaning federated search mash-ups that included content from servers that I run.
Filtering by useragent seems to be a temporary solution. I think that's especially true since you've disclosed publicly that's what you're doing.
I had problems with the Windows updater apparently downloading version 2.0.6 and then not installing it. I ended up downloading and installing it manually instead. Could that be related somehow?
Why do you return 403 (according to the article) instead of 444? 444 will drop and close the connection, so I am guessing it has something to do with keep alives vs connection overhead?
If you like logstalgia, you'll love gource[1] which visualises source control change history. It's also available as a brew recipe on the Mac for easy-peasy install.
As someone who over the past decade regularly had to sell higher management on investing in infrastructure, I can tell you this will be extremely useful.
Seriously, even if it's just a random visualization of some normal peek traffic, it will go down easier than dozens of actual arguments, slideshows with bullet points or fat reports nobody reads. Show the video, do a bit of handwaving and there's your budget...
For any techie trying to convince their boss they need more servers, this is a very powerful tool. Use with caution.
So I installed the project and ran it with my log files and I'd like to retract my previous claim of it not being useful. You're right, I can definitely see the use here, it puts stuff into perpective quite well.
It's easy to write off shiny visualizations as silly (which is what I did above) but they definitely offer a perpective that you can't get any other way.
Perhaps it's worth it to code a quick and dirty solution using JavaScript encryption. On your download page setup a script that would receive a given encrypted string, decrypt it with a provided key, and the use it to prepend to the download link. On the server, symbolically link the file on demand and send it to only one user, ip limited. This way the attack, though still can be automated, would require some code rewrite from that attacker, which might be beyond his/her abilities. Also, if the encryption algorithm is CPU intensive, then it wold require several seconds of CPU time per request from the attacker.
To make the decryption CPU intensive you may simply use any encryption algorithm you like, many are available as JS libraries, but instead of giving the entire decrypt key, skip the last 2 digits, and let the end user brut force the last 2 digits in the client via JS. That way there is a computational cost to each attack request.
Just some ideas off the top of my head. Not sure at the moment how to implement the server side part at the moment, but I am guessing that their are server side rules that allow you to easily set per ip access restrictions to folders or fils.
PS: please excuse spelling, typing this on my iPhone.
I'd be willing to go out on a limb and estimate that maybe some private interests in Hollywood, with certain four letter acronyms, despise open source media player projects like VLC, since they might represent a channel that can potentially enable bypasses that can circumvent precious, precious DRM.
The perception being: if you can see the source of a media player program, the encryption might be implicitly compromised. This is a silly idea though, because it neglects certain realities about the very nature of electronic encryption, and media consumption. Maybe having source code lowers the bar in some respects, but the reality is that determined people will simply bootleg media anyway, by other means.
Not an accusation though, just that my tinfoil hat is tingling. Who else might be so motivated to attack an awesome software project like VLC?
Cool / "good" projects get attacked all the time. I suspect this is because the attacker is guaranteed an audience on HN to marvel in the results of their work. I think this is probably simpler and more likely than the tinfoil hat theory.
Or, less likely but more timely, the GPL→LGPL relicensing to get an AppStore port, or the consultation to get the French DRM police to play its role in enforcing a conflicting law about interoperability.
> Who else might be so motivated to attack an awesome software project like VLC?
Anyone who wants to show off a proof-of-concept attack to potential customers, without stepping on any huge toes. (e.g., it's relatively safe to DDoS VLC, not so much a U.S. government agency or a huge multi-national company.)
Why DDOS Reddit? Why DDOS Wikipedia? Why DDOS the New York Public Library website??
Within the last year or two I've given up trying to find reasons for DDOS attacks. I think of them now like Internet traffic weather. They just happen at random times and you better be prepared.
> The actual number of requests was not that high (400 req/s), but the botnet was downloading the whole vlc.exe, aka 22MB. So, we were at around 70Gbps during the night, in average.
Source spoofing would not allow downloading the whole file; the spoofed source address would get the first response packet and send an RST ("stop, no connection associated with this packet, go away") long before that point.
It it possible this is an accidental DDoS? VLC is popular to bundle with things, and all it would take is the code that checks for a new version and automatically downloads to have a bug that it always thinks there's a new version...
Ubuntu downloads from it's own mirrors (hundreds of them), with very few exceptions for proprietary stuff that can't be distributed like Adobe Flash and ttf-mscorefonts-installer (not default). Also it would not download a .exe to install VLC.
I actually have Logstalgia running with my primary server for Minotar, and at 4,000 requests per second this is normally what it looks like. Awesome program!
In my experience it is fairly manual. Here it is at a very high level. First you want to determine if this is really a DDoS or legitimate traffic.
You might be notified via downtime, alerts of load, in this instance, I suspect; download graphs, log analytic (if using a cdn which can handle the load, then you might not notice for a while i.e. eye popping bill).
Narrowing down the attack profile means looking at logs. Be that network flow data (very helpful) or in this instance web server logs. Probably something like: totals grouped by ip, destination url, etc to see if there are any spikes.
Also, managing stress. If you are some type of retailer then you likely are losing money, people are asking for updates, etc. This can be extremely stressful.
We use Munin for the graphs and Nagios for the alerts. The mitigation of the DDoS is done manually since we need to identify the patterns and block them.
I currently have a native OSX version og glTail in review by Apple, which is quite a bit faster and easier to install than the Ruby version. I also have an updated IOS version in the queue which doesn't crash quite as often.
Fantastic! I had read about such a log visualisation tool a long time ago (I'm not sure but I think I read it about it via NTK which should date it) but I had lost any knowledge of what it might be until now.
I just got one of my servers attacking TicketMaster by a faulty cgi.
(my alert system notified 5 minutes after it started)
The mob is angry now It was disabled..
I think it has more targets that only vlc...
I was browsing HN on a friend's computer that without adblock and clicked this link. Wow! Is this what the internet looks like without adblock? The ad/content ratio is crazy...
The attack started on our new mirroring system (powered by mirrorbrain) 2.5 days ago, during the night (after 2am, so we were sleeping).
We were woken up (OP and I) in the morning by many mirrors complaining of high bandwidth use. The actual number of requests was not that high (400 req/s), but the botnet was downloading the whole vlc.exe, aka 22MB. So, we were at around 70Gbps during the night, in average.
Afterwards, North America got up, and things got worse. We had up to 1660 req/s, so around 292 Gbps...
This is very weird for a DDoS, to be honest.
Our front machine that splits the down mirrors was taking most of the load, and we were able to find the patterns to drop the botnet connexions, in order to not kill too much our mirrors. I won't discuss too much of the patterns, as you can imagine, but as usual, I'll be happy to discuss it IRL or by mail.
Tweaking the front server was also important to reduce the number of open connexions, to not kill our server.
2.5 days after, the attack is still going on, with an average of 500req/s.
The video was done using logstalgia, using scripts of OP, on my machine (<troll>he was running eclipse, he couldn't do both at the same time :)</troll>).