To me the warrantless wiretapping scandal IS the proof.
I mean if we know that there are Room 614As [1] all over the country, and that they are diverting nearly ALL communications to their facilities, in addition to them building new facilities to house data all the time [2].
All of this culminating with their shiny new data center [3], which cost 2 billion dollars to build, covers 900,000 sqft, and has a 40 million dollar annual utility bill, it becomes less about proving that they're doing this, but proving that they aren't doing this.
Even at 100k square feet, I don't think they have the capacity to record and store all data on the Internet backbone (they're recording a lot of data outside the US, too).
First, they'd need a shadow Internet backbone capable of transporting all that data to the DC. Second, they'd have to build a system with the same write-rate as all of the internet backbone. This stuff isn't cheap, and the NSA's budget is big, but not that big.
As 1/3rd of the US's internet traffic is porn, which you can really just de-dup by URL, it's a safe bet to say that they're filtering before they transmit data back home. The same goes for streaming video traffic. The remainder is mostly web with a small fraction being actual communication between people. For the NSA's mission, even implemented in the most evil way, they just don't have the money, means, or motivation to record everything. Instead, filtering on *hotmail.com connections, anything over SMTP ports, etc. makes substantially more sense.
Doing some digging, I find it rather hard to find data on the amount of traffic being sent in and out, and internally of the US. The best numbers I could find was from almost 10 years ago, which said around 970Gbit/s. Say it has gone up 100x, and we get a nice round number of 10 TB disk space needed per second.
We know, thanks to XCD, that 7523 hard drives per second is created by the storage industry (globally, a total of 650 million drives per year). Say that the average storage space is over the last 5 years, around 500GB.
That mean, so long NSA buy's 0.265% of all hard drives produced each year, they will have enough hard drives to record all data transmitted inside the US borders.
They would still need to write the data. One obvious way would be to store it on site, and transport the drives to a central place. Drives are not big, but it is a noticeable work, so if people were doing this, there should be more verifiable proof of it. If we include post-storage compression, finding duplicates and any other tricks, the numbers should be able to be lowered by 50-75% or so, and might be enough to send some through the wire and only the overflow through drives loaded onto trucks.
One could also ask what 0.265% of the storage industry output is in raw cash. To answer that, my answer is, I dont know :).
There could also be an incoming relevance filter. I'm sure the NSA has no interest in archiving all the pornography, spam, and cat pictures ever transmitted.
Also note that the majority of bandwidth nowadays is spent on videos and similar multimedia content. So if they notice that 100M people are watching Bieber's latest music video, they just need to store the YouTube URL. No need to store 100M copies of the video itself.
They don't need to store the traffic. All they need is to store the endpoints and timestamps and you could compress that quite well to have usable information, such as who talked to who and when.
Your actual data is just a subpoena away anyway in most cases.
Money are of no objection to that industry. The combined budget of NSA, CIA, FBI, Pentagon, foreign bases, Afghanistan and Iraq occupation, etc. is in the trillions. Why trust FB or Google if all it costs to capture and store the information are money.
NSA is an intelligence agency, not a law enforcement agency. They don't operate within the court process. Which means they don't really do subpoenas or warrants. Furthermore, any data covertly collected by the NSA would be inadmissible in court anyway, which would make it impossible to get a subpoena or warrant based on it anyway.
In 1971 we put wire taps on Soviet underwater communication lines within territorial waters despite sound detection devices placed along the seabed. I'm sure the country that had the ingenuity and balls to pull off Operation Ivy Bells (http://en.wikipedia.org/wiki/Operation_Ivy_Bells) when it faced the existential threat of Communism is perfectly capable of figuring out how to do the exact same thing to packet communication in the continental United States as it faces the threat of Terrorism.
It is a fundamental issue of volume. Unless we assume that backbone providers and Internet companies habitually over-provision, then it would take about the same size pipe, and the same size data centers, to "copy" the Internet in real time.
Consider how big the Internet is. Even if the NSA has 5 100,000 square foot data centers, think about adding up the aggregate data center footprint of Microsoft, Facebook, Apple, Google, Yahoo, Amazon, Rackspace, etc, etc. That comes to a lot more than 500,000 square feet. And this does not even get into the enormous data resources in Asia and Europe.
A much more likely scenario is that they are heavily filtering the data in real time and keeping just what looks useful or suspicious. That is still scary, but less fantastic than the idea that every packet is getting stored by the NSA.
I've looked at my own traffic and how much content useful for storing I'd have and it's probably less than 500Mbytes per month. And that's counting downstream and upstream.
But net is not the only thing NSA is interested in storing - banking, other records, communications over satellites, etc goes in there as well.
What's truly scary is people are sort of "meh" about this. Or they don't believe that this is really happening, as most of this thread seems to be the case in point.
The porn is in the top ten of their priority list. A great way to find the weak points and blackmail possibilities for current and future opposition leaders.
An interesting take on your comment: would people be more open to the NSA snarfing everything if they also acted as a personal Google, giving you access to all your own "lost" data?
I've been thinking about this lately, with regard to my own privacy policy.
I was thinking something to the effect "I will make all data I save that references your account or IP available to you through the following portal" - I of course, can't do it yet, as I haven't built the portal or figured out how to separate the data, but I think it's a neat solution to some of the problems I have (e.g. I really want to start doing console logging on VPSs 'cause it gives me a wealth of data when someone says "something was wrong with my VPS last night" - but so far I haven't (I do log physical consoles, just 'cause it's impossible to deal with hardware otherwise) due to privacy concerns.)
at /domain/portal just put a contact page and ask them to specify what they are looking for. You'll have to do it manually until you build the automated system but maybe the inconvenience would be motivation?
I need to limit the number of times I give myself extra work as 'motivation' - you get to the point where you spend all your time shooting alligators, and don't have any time at all to drain the swamp.
I mean, sometimes this is better than just not doing it? but not in this case, if you ask me.
Considering how quickly e-mail privacy is starting to get into lawmakers agenda after Patreus' fiasco I think this would have great consequences. Of course this implies nsa being open in return - that won't happen.
Somewhere I read a quote: People under surveillance, are already imprisoned.
The first step is to understand that one is under surveillance.
I mean if we know that there are Room 614As [1] all over the country, and that they are diverting nearly ALL communications to their facilities, in addition to them building new facilities to house data all the time [2].
All of this culminating with their shiny new data center [3], which cost 2 billion dollars to build, covers 900,000 sqft, and has a 40 million dollar annual utility bill, it becomes less about proving that they're doing this, but proving that they aren't doing this.
[1] http://en.wikipedia.org/wiki/Room_641A
[2] http://www2.sacurrent.com/news/story.asp?id=69607
[3] http://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/al...