Certainly it would be easier, but even NSA doesn't have enough hard drives to store all of GMail, Outlook.com, Skype, etc.
Some selectivity is required for them to do analysis, for the same reason we would run "grep foo <star>.cpp" instead of "find / -name '<star>.cpp' -exec grep foo {} \;"
> Certainly it would be easier, but even NSA doesn't have enough hard drives to store all of GMail, Outlook.com, Skype, etc.
Actually, that's an interesting point. I don't bother keeping copies of television series I've downloaded and watched laying about on my disk, because I know I can just go and retrieve them from the internet again if-and-when I want to watch them. In effect, I'm using the internet as a large, slow (but reliable!) disk.
The parallel being, I don't see why the NSA would bother to build its own datacenters for storing data. Google's, Apple's, Microsoft's, whoever's datacenters, are the NSA's datacenters. They don't have to retain information themselves; they just have to send a FISA notice to these companies telling them to retain the information, indefinitely, until the NSA has need of it.
> The parallel being, I don't see why the NSA would bother to build its own datacenters for storing data. Google's, Apple's, Microsoft's, whoever's datacenters, are the NSA's datacenters. They don't have to retain information themselves; they just have to send a FISA notice to these companies telling them to retain the information, indefinitely, until the NSA has need of it.
That's something I noted when this all blew up, was that simply keeping the NSA from getting the data was not good enough, as they could simply compel the phone company (or whoever) to hold onto it, and that we needed to be ready for something like this to be close to the "new normal".
However it could still be better to go this route, in that even though the NSA can get metadata on everyone that's 3 hops or thereabouts away from the phone company, that's still not a huge mass of data. And the phone company could notice if there was a large number of requests being made, do random audits to verify that NSA's Compliance office approved that given search, etc.
But then you would have to ensure those in the phone company with access to the logs for that program themselves have security clearances since they would in a very real sense be monitoring the progress of national security investigations (imagine if you had a sysadmin that was an AQ sympathizer noticing the NSA has penetrated a terror cell and he manages to tip off the cell in time).
Certainly it would be easier, but even NSA
doesn't have enough hard drives to store
all of GMail, Outlook.com, Skype, etc.
Gmail has enough drives - why shouldn't the NSA?
Gmail offers 15 gigabytes of storage space, almost certainly over-allocated - I know only one or two people who have hit that limit, personally I'm at less than 3 gigabytes. Gmail has 425 million users, according to [1]
According to Wikipedia's Utah Data Center article[2], it's estimated to have 3 to 12 exabytes of storage in the near term. So, space enough to store 1.0 billion to 4.2 billion 3 gigabyte gmail accounts, or 7.6 to 30.3 gigabytes per gmail user.
Somewhat short of 'all internet traffic all the time' but they could get all of gmail easily.
Some selectivity is required for them to do analysis, for the same reason we would run "grep foo <star>.cpp" instead of "find / -name '<star>.cpp' -exec grep foo {} \;"