Benford's law does not apply to some of those things! In particular not to expenses.
Benford's law only applies to things that experience exponential growth, things that obey a power law distribution, and things that are totals of random processes.
How could expenses not have a power law component to their distribution? Sometimes you're buying an expensive thing, and sometimes a cheap thing, and you don't buy something between $999 and $200 dollars nine times as often as you buy something between $199 and $100.
Yes, you will certainly get artifacts at the top of what you can expense. You will probably also get distortions at the bottom where you'll have some threshold of "not worth expensing." There will almost certainly be distortion around the boundaries, but as long as you have a big enough range you should be able to see a Benford's law effect in the middle.
Benford's law would not have stopped the groupthink, willful ignorance and corruption which actually drove the Greek disaster. It's not like Greece's problem with tax collection was a well-kept secret, it's just that nobody wanted to face the facts.
I seem to remember at the time there was widespread reported skepticism about a number of countries' data. It was just accepted that the central European elite wanted to expand to sell their wares to these dodgy countries and prevent conflict. It shows how dependent the lender is to the lendee and how both sides don't face up to the reality.
Also, Greece’s original accession to the EU was motivated in part by geopolitical considerations; Greece applied for membership shortly after a democratic government replaced seven years of harsh military rule.
The issue with using Benford's Law with analysis is that as soon as people realize that you're using it, data can easily be manipulated to make it compatible.
The previous article by Tim Harford mentioned that Madoff's financial data was Benford compatible.
So I doubt that using it would have actually prevented the Greek or any other disaster.
The article says the opposite, though it's hard to put a lot of credibility on something starting with "according to statisticians"...
According to statisticians, it is almost impossible to manipulate data in a way that a certain outcome is guaranteed and Benford’s Law is met at the same time. Hence, tax authorities in several countries are using Benford’s Law as a default testing device.
I re-read the article and you're right. The article seems to imply Madoff as an exception rather than the common case, but I still feel that if data is being manipulated at such a vast scale where an entire countries economic activity is misrepresented, you will be able to find people as smart as Madoff to 'fit' the data as per requirements.
You certainly can find people smart enough to make sure that their fraudulent statistics will meet Benford's Law (or any other data model you wish). The article points out that Greece didn't find people this smart.
Not meeting Benford's Law is a good indicator that there has been some dodgy dealing. However, meeting Benford's Law is not a good indicator that everything is above board.
Nice geek porn, but the title and the premise is basically nonsense.
EU politicians knew for many years that Greece was scamming things. Maybe not so much when the Euro started, but well before it all blew up. All kinds of circumstances, however (such as the need for Greece's vote on some other hot issue of the moment) made sure that no one had the will to really do something about it.
Statistics is nice and dandy, but it doesn't help a dime in fixing the broken mess that is EU politics.
ps. Fun trivia: if the EU applied for EU membership, it wouldn't get in because it's too undemocratic.
It's nice that they made sure the data set contained the key data to their collapse. The pensions of retirees. Would the results have been the same if it weren't included? I don't have access to the paper but I'd be curious what their justification for their choice was.
Among other things, they looked at the total level of debt, the cash reserves of the government and the pensions of retired civil servants.
Not sure if you skimmed the linked article and missed this or not:
statistics from the Czech Republic, Sweden and the UK meet the Benford distribution particularly close and hence appear unsuspicious. All those countries have no interest in joining the Euro and therefore do not have to meet the convergence criteria. On the other hand, statistics from Latvia, Belgium and Romania (the latter notorious for its corruption) appear very suspicious, as well.
"They looked at 130 different values per country and year", they don't say how many years were studied but I assume the greece book fixing lasted only a few years so we are looking at ~500 samples. I honestly don't know if that's enough or not, on wikipedia they show examples of the distribution on less than 100 samples so I guess I'm on the wrong.
> According to statisticians, it is almost impossible to manipulate data in a way that a certain outcome is guaranteed and Benford’s Law is met at the same time. Hence, tax authorities in several countries are using Benford’s Law as a default testing device.
I'd be very interested in evidence to support this. This doesn't sound correct. "Almost impossible?"
Although in this instance I agree, I think the bigger point is that you shouldn't call anything "arcane" on HN. Because it'll be well known to someone and that someone—who is probably someone who could contribute a lot to the HN discussion—will think poorly of you.
Come on, this is not even a real law, and the data are really shaky to justify the title of this post. 5% significance threshold means that Luxembourg or Austria satisfy the criteroa. So what? The EU is a large political organization, not a NYSE index. The real trouble for greece is not the fake accession data (Italy had similarly manipulated data), but the squandering of public money that followed.
I get it, geeks love to play with empirical laws, but this paper is at best interesting trivia. I could go on analyzing what could really prevent the disaster forever ...
1. Spotting odd things in MPs' expenses: http://blog.jgc.org/2009/06/its-probably-worth-testing-mps.h...
2. Spotting odd things in BBC executives' expenses: http://blog.jgc.org/2009/06/running-numbers-on-bbc-executive...
3. The Iranian election: http://blog.jgc.org/2009/06/benfords-law-and-iranian-electio...
4. New Age mumbo jumbo: http://www.jgc.org/blog/2008/02/any-sufficiently-simple-expl...