Building a haystack to search for needles isn't free. If it's not delivering enough needles then people are going to be under pressure to end the program. I think a lot of people in government have smartened up to the fact that these things can be just as easily used against them if they fall out of favor and that having this sort of ongoing data collection could become liability if the wrong kind of people are in charge. The kinds of things Muller has chosen to prosecute certainly makes a lot of people in Washington nervous about their skeletons in the closet so there's less political tolerance of these kinds of programs than there once was. Snowden's leaks and greater public knowledge of these programs have reduced public support. It frees up resources to go after Russian trolls (or whatever), hunting terrorists is so 2005. There really is no way you can justify a program like this in 2019 (or so we hope), they've been proven ineffective and unpopular.
It's not really a needle in haystack problem. If you record every moment of all your citizens. You can build a calendar application where you enter the name of the person and go through all the moments in question throughout the days of curiosity. Specifically when that person is under suspicions of some criminal involvement. I feel sorry for china because I can only imagine how much more data besides just phone calls are collected from citizens unaware.
Let us hope that you are correct. We would still need to fortify laws against data collection that is fiercely enforced (although already being a requirement in many constitutions in the western sphere).
That would require oversight with severe consequences for any program breaking the rules. Personal consequences up the chain. Intelligence agencies have morphed to petty criminal gangs. Even if you can easily argue that they have been criminal before...
Considering congress's general lack of interest in providing thoughtful oversight, it seems they could just setup another program and congress would just shrug.
I think a lot of people at the NSA are relieved to not be doing this anymore. These programs were started post-9/11 when we had no idea whether they would be effective but we felt like we needed to try them or at least try something because they might be the silver bullet that prevents another 9/11. People knew they were constitutionally dark gray and fundamentally un-American on day 1 but they went along because it might help. Many of the more senior people in the early 2000s were old enough to recall the Hoover era and everything that came to light after it. Now that we've proven bulk collection doesn't work we can finally put it to rest. With the increasing popularity of encryption it was getting long in the tooth anyway.
I also think that the established political parties are realizing how much power truly does lie in the executive branch and how long four years really is. While the current example being used to teach these lessons may be highly repugnant to the current political elite (on both sides) he is not doing anything that cannot be undone relatively easily.
The bigger the program the shorter the time it will stay secret, a nation scale metadata collection program is necessarily big. Congress will find out about it eventually at which point they will not be happy (Feinstein and a few die hard authoritarians excepted) because said program is all downside and no upside (as proven by PRISM's track record).
Nobody in Washington wants a time bomb like the calendar use case you described sitting around lest some future far left or right president use it in a manner that inadvertently starts a civil war.
That said, I also fully expect that they've moved on to some equally nefarious shit without telling us because that's what intelligence agencies do.
> Nobody in Washington wants a time bomb like you described sitting around lest some future far left or right president use it in a manner that inadvertently starts a civil war.
A large segment of the american population already strongly believe that the whole post 9/11 spy apparatus has been abused for political means. Even if these beliefs are unfounded, the existence of a system that is ripe for abuse has eroded trust in American institutions so far that I'm not sure that the country will ever recover. Belief in Justice and trust in checks and balances is as important as the implementation.
