> To discern the system's true purpose, I used Masscan to ping TCP port 80 across the entire IPv4 address space using a TTL of 10 [...] Sampling this list revealed that the system was primarily being used to block access to copyrighted content.
Yes but he also mentions it originally existed to block child pornography, and has been repurposed to block copyright infringement. Since mail.ru is neither of those, it has obviously been repurposed for something else as well. But what?
Sure, that's indeed possible, but I'm not sure it's "obviously" a sign of re-purposing. It could just as easily be because someone was sharing warez from a common mail account, or some content owner erroneously reported the web site (which has definitely happened in the past!); either way, bam, now it's on a blacklist forever.
If that's the case, then the burden of proof for copyright-based censorship is questionably low in this UK scheme.
I mean, mail.ru clearly has a primarily non-CP non-infringement purpose; banning it (is it actually banned?) should have been transparently disproportionate to anyone reviewing that claim.
There must be something we're missing. Not that I think the tin-foil-hat theories sound much more likely.
About 10 years ago the CP block list used by multiple nordic ISPs were leaked and people jumped on the opportunity to test the quality of those listed. The result they got where that a couple percent was CP and the remaining was either dead or non-CP.
The problem (as was discussed in Sweden during that time), is that the list can't be made transparent since according to the police that would "guide" people to location of CP, and the owners of website is not contacted since thats not the job of those operating the lists. There is also no one willing to employ and pay people to clean the list of domains once the content is gone.
In one particular case there was a Bonsai website that got listed. Those studying the leaked list called the owners of it and the owners were completely unaware that most people in the nordic countries could not access their website since it was blocked. The Swedish police speculated in a interview that the site had been hacked and that the operators (web host/designers) cleaned it without notifying the site owners.
As a administrator for web hosting, I would be amiss if I didn't add to this that hacked websites have generally a multiple of payloads at the time anyone will start notice that something is amiss. A site could be sending spam, selling fake shoes, and running a CP site all at the same time, and the operators could have notices one of the payloads then decided to reinstall (and update).
The "CleanFeed" technology mentioned in the article, and similar technologies are deployed at all major UK ISPs (basically if you've seen it advertised on TV, it has this) which was, as the author says, deployed originally to intercept potential "Child Pornography" and block it using an otherwise transparent HTTP proxy. The proxy looks at the HTTP 1.1 Host header, or even the entire HTTP request, and decides either "Yeah that's bad" and gives a bad saying you're a bad person and should feel bad or lets it pass unmolested. This is impossible for HTTPS because the server's identity is authenticated using public key crypto
Anyway, this illustrates a peril of developing technology. Inevitably Big Copyright (in this case I think originally the major music labels) took these ISPs to court. The court asked them a plain and simple question - do you have technology that could do what Big Copyright wants here, and block arbitrary sites? And CleanFeed meant the answer was "Yes". So the court said, "OK, since you _can_ do this, we're ordering you to do it".
Now, some small UK ISPs simply don't have censorship capability, and the court told them since you don't censor anything (Child Porn, videos of people being gruesomely killed, how to cheat at popular video games) we don't find that you have any liability for not censoring copyright infringement either, carry on.
Years later this got extended still further, the UK government "ordered" the big famous ISPs to censor stuff that might offend people or was "inappropriate" for children or whatever. And all of this is managed opaquely so it can be used for any purpose that the government sees fit. But again, small ISPs simply don't obey, and the UK executive has a _history_ dating back to when printing books was a new technology, of going to court, insisting that "public morality" demands censorship and getting shut down very embarrassingly by judges, so this time they have chosen to simply pretend the smaller ISPs don't exist. tl;dr if you live in the UK and people have heard of your ISP your access to the Internet is censored. Switch to a tiny ISP run by people who'd rather face jail than destroy the Internet, or pipe everything through a VPN and/or Tor.
Note that this is all separate from the UK "Snooper's Charter" which is very secretive but probably implemented in the backbone. The "Snooper's Charter" stuff doesn't block anything, but does monitor metadata for everybody, all the time, basically like the NSA in the US. Changing ISP can't help you there.
This is both fantastic discussion of the ability to quickly check for vulnerabilities in ways never possible at scale before and also discussion of a little-known vulnerability that even the security experts were not aware of.
The security guys at Imperva Incapsula just wrote up how they protect their system against this here (https://www.incapsula.com/blog/http-host-header-fix.html) – and in their tests, the only vulnerability they found was their own tests. But they wouldn’t have done it until the BlackHat presentation.
Security in depth. Make things secure, even if it's behind a firewall. Start by searching your LAN, and watch out for open shares, weak or default passwords, and unpatched systems. Configure every machine like it's on the public Internet.
SSRF is going to be the major story for the next couple years. I've seen them all over the place and they are much harder to block since they exploit multiple protocols and the properties of URLs have many arcane rules that aren't known to most programmers.
Reading this a while back really opened my eyes to how complicated it was, even if I'd already learned (the hard way) most of it:
I just uncovered (by accident) and reported an xss vulnerability in a service once used by the Obama administration, and Obama himself, and have reported 3 other vulnerabilities I've found to other companies, one was to Google. Didn't get any money for any of them (Google sent me a Nexus 7 tablet, which was nice). :-(
Look like I might need to start using Yahoo more! Haha.
This was a great read. To others: make sure you are allowed to pentest a server before doing it.
