I will look into brad delong's blog later when I have time, but I did factcheck some of shadowstats numbers four years ago for work, and they were right on the money.
E.g., the modern definition of unemployment is indeed very different from the definition of 40 years ago, which would yield something like 20%-25% today (compared to 12% or so today). And yet, everyone assumes it is the same statistic. Same with inflation.
Personally, my 3 biggest expenses (rent, food, energy, in that order) have been going up at almost 10%/year over the last 3-4 years. I don't see a reason to doubt the shadowstats CPI metric.
You're welcome to trust BLS data. Just be aware that you can't trust them when they are published [ http://www.zerohedge.com/article/visualizing-propaganda-erro... ] and given that 90% of the values are revised to be worse within a month, their methodology cannot be considered statistically sound.
edit: added "almost" to 10%. Just checked, and rent was "only" 8%/year.
Implying that most other economists in the US are dishonest is a bold claim that needs some significant evidence.
Implying that something is dishonest because it is different than it was in the past is a non sequitur.
"Personally, my 3 biggest expenses (rent, food, energy, in that order) have been going up at almost 10%/year over the last 3-4 years. I don't see a reason to doubt the shadowstats CPI metric."
What was that you were saying about sound statistics ;)
> Implying that most other economists in the US are dishonest is a bold claim that needs some significant evidence.
I do not think that, and I do not think that I implied that anywhere.
Rather, almost no one outside the BLS collects this information in the US on a scale that allows you to contest official numbers - so that BLS gets to claim whatever they want, and most economists in the US take that as gospel (and are wrong because they are misled, not because they are dishonest).
> Implying that something is dishonest because it is different than it was in the past is a non sequitur.
Appeal to authority ("obviously, the BLS/Fed had their legitimate reasons to revise the calculations") is a fallacy. And I did give the reasoning, so let me repeat amd expand that in case you missed:
Government gets income indexed to real inflation (by virtue of being a percentage of wages/profits/transactions) - so it is not affected by accuracy of inflation numbers.
Government expenditure (salaries, social security) is indexed to a computation controlled by the government. Furthermore, if this computation results in a number 0%-3%, everyone is happy, even if this is not supported by the underlying facts.
Said computation has been revised several times in the last 40 years, where new computation always and consistently results in a number lower than it would have been had you used earlier formulas.
Now, what do you find more plausible: Everyone in government (and extended government - the Fed) was trying their best to understand the economy, and it just happens that all revisions make government look better - or that these revisions were made because they make government look better, despite underlying facts?
Oh, and take into account that government and the fed is actually full of career politicians (bernanke campaigned to become fed chief).
At some point I factchecked government stats and actions for a living. My conclusion is that it is all fudged, nothing can be trusted, mistakes get made to make the government look better 90% of the time. Law enforcement (especially in case of the SEC) is very selectively applied. I can cite numerous examples, but I'm tired. Do your own research (karl deninger, zerohedge, michael shedlock are good starting points), or keep watching the CNN / CNBC government mouthpieces and believe all is well.
Is the BPP index faked as well? Is krugman dishonest or mislead, or maybe incompetent?
This can also be a difference of opinion, that we can settle with lot's time. But I still dislike your original implication that people asserting a different number for inflation are dishonest.
I cannot decide whether Krugman is dishonest or incompetent, but way too many of his assertions and op-eds can be factually contradicted; deninger used to do that weekly until he got tired of it.
Did you notice that between 1-Jul-2010 and 31-dec-2010 the BPP was >=2.3 (closer to 3% most of the time) while the official CPI was at <1.3% (closer to 1% most of the time)?
The BPP was constructed to be similar to the CPI. If you constructed it to be close to the CPI of 30 years ago, it would have been higher.
> But I still dislike your original implication that people asserting a different number for inflation are dishonest.
It's not arbitrary people. The number of people who decide the CPI formula is ridiculously small - there might be 20 economists working on it, but ultimately it is a political decision by a handful of politically motivated people at the BLS to switch to a new computation, and the entire economic profession just accepts it as gospel.
And I stand by my assertion that these politically motivated decisions do not have any "scientific purity" (whatever that might mean) or the people's well being in mind -- it has specific goals in mind, which end up harmful (through no direct malice)
I read it, the problem is that you imply things rather than stating them directly. I can't read your mind, so I don't always get what you are implying.
Example: You assert that the CPI is essentially fraudulent, and chosen for political gains and that their is a conspiracy afoot. I point out that the BPP shows numbers much closer to the CPI than to shadowstats. You reply that it's constructed to be similar to CPI. This implies something, right? It either implies that it's part of the conspiracy, or the people are incompetent, or something ... but I don't know what your point is. Also, your point doesn't mean that it's wrong.
Also, can you please show some evidence that the BPP was constructed to be similar to CPI. Right now we have 3 sets of numbers, two of them roughly agree. I'm inclined to believe that the BPP and CPI are more likely right than shadowstats given their agreement and the fact that they use different data sources.
> I can't read your mind, so I don't always get what you are implying.
I try not to imply anything, but I do assume that you have some background, formal or informal, in statistics and economics (if you don't, I'm surprised by how strong your opinions are given that you would be missing the tools to evaluate them).
For example:
> can you please show some evidence that the BPP was constructed to be similar to CPI
CPI and BPP are both a weighted average over a list of prices (several thousands or millions of them, with CPI averaging only a few carefully selected hundreds, and BPP supposedly averaging a few tens of thousands or millions).
Statistically, it is impossible for these two averages to be in such close correlation as they are, unless: a) essentially _all_ billion numbers sampled from have the same exponential evolution (that is, they all increase by 0.1%+/-0.01% per month, every single month) -- which is demonstrably false (oil prices have gone up and down several times last years, food prices have only gone up), or b) these averages were constructed to track each other (or a common underlying, but the CPI is defined to track some abstract underlying which is often revised and not perfectly documented or repeatable, so it has to be that BPP tracks CPI).
The probability that the BPP index was independently constructed and yet tracks the CPI so low that one would need evidence to believe it. That's how statistics works, and you're welcome to consult a stats person if you don't believe me.
What I'm saying is not that they are trying to do the same thing that CPI is doing. What I'm saying is that they did a regression of the prices they collect against the CPI officially published CPI, and as a result, they have a good approximation to the officially published CPI, without ever having to have an opinion of whether the official CPI is a good way to gauge customer expenditures.
> You assert that the CPI is essentially fraudulent
Yes, I assert that, as a measure of the growth of consumer expenditures, it is wrong (in the sense that it is inaccurate), and that it is fraudulent (in the sense that it is underestimating the price increases, and that it is intentional).
> their is a conspiracy afoot
These are your words, not mine. The thing is, your belief that there needs to be a large number of conspiring people in order to mislead others (intentionally or unintentionally) is not supported by evidence. Take medicine for example - for tens of years, long after the "evidence based medicine" came around, 99.9% of doctors (scientists! with diplomas!) believed stomach ulcers were caused by stress. Warren's claim that it is caused by a virus was considered craziness to the point that he had to infect and heal himself for others to take him seriously (later earning him a nobel prize in medicine). A recent survey showed that 40% of US doctors still believe the ulcers are caused by stress, despite never seeing any evidence for that and having evidence that it is caused by a virus.
Now, how do you explain that? Were all those doctors, 99% of practitioners in the US and elsewhere, dishonest? No. Just misled. And I can assure you it didn't take more than 5 people a century or two ago to establish that "ulcers come from stress" for that to be accepted. Do you see the analogy? [And if you find that interesting, you're welcome to ask doctors, and then look for evidence, that salt intake increases your blood pressure, or that cholesterol intake increases your serum cholesterol, or that serum cholesterol is bad for you for that matter. You'd be amazed at what doctors believe without any evidence, and often evidence to the contrary. And they are not dishonest, they are just misled]
> This implies something, right? It either implies that it's part of the conspiracy, or the people are incompetent, or something ... but I don't know what your point is. Also, your point doesn't mean that it's wrong.
No. It does not imply anything other than what I said: it was constructed to track the CPI (the officially published number called "CPI", not anything else). It is not wrong in that sense that it IS a good approximation to the CPI, derived from independent other sources. And it does not make anyone who works on it part of a conspiracy, or incompetent. But it does mean they accept the CPI as gospel (which I explicitly claimed). Is there any reason to believe the BPP tracks the average growth of consumer expenditures? (averaged over consumers, of course)?
> I'm inclined to believe that the BPP and CPI are more likely right than shadowstats given their agreement and the fact that they use different data sources.
As I mentioned, from a statistical point of view, these can only agree if they are constructed to agree, so they shouldn't be considered independent. If you want the economic reasoning for that:
The CPI switched in 1983 from including real estate home prices to using "owner equivalent rent". While one may argue whether this is justified, it definitely changes the meaning of CPI in many more ways than you'd expect (e.g. home prices were based on sales, OER is completely speculative). If you kept the old computation for just this component, this is how 2000-2010 would have looked: http://seekingalpha.com/article/45720-how-owner-s-equivalent... (which is much closer to shadowstats - and remember, this is just reverting one component back to 1983 computation, everything else the same); If you look at the BLS faq http://www.bls.gov/cpi/cpiqa.htm#Question_2 , they quote NAR data to show the BLS are saints. If you actually follow the NAR data, you'd be laughing at using it as reference.
If the BPP used house prices, things would look very different. They cannot use speculative OER at all because only the BLS have that data. The fact that BPP agrees with CPI indicates that they try to track it, rather than figure out a true "basket".
Furthermore, even if you take CPI as gospel, the BPP has 1% difference over 3.7 years it exists. Assuming that this difference remains, in 50 years, your government adjustment benefits will buy 15% less than what they officially should be able to buy.
If you are really interested in challenging your world view, read Karl Deninger's blog for a month (the market ticker). Stuff you'll never see in the New York Times, CNN or CNBC.
You keep questioning my statistics skill, but you are the one who used the anecdote of your personal purchases as evidence. Your analogy makes sense, but again, you cherry picked one data point. For every case like this, there are 100 where the professional consensus was right, and the lone dissenter was simply a quack.
Of course, there is a giant different between your analogy and what we are debating. You guys haven't proven beyond a doubt that inflation is significantly higher than the BPP or CPI. You are also accusing BPP of fudging their numbers, and admittedly they aren't open about their data sources, but here's the rub.
You could easily create your own BPP type index, make it open with the code in the public domain, and show some real numbers to prove that the BPP is fudged. Right?
side note: shadowstats is not transparent either. They haven't produced the data sources plus the code that generates their numbers.
I have not done that anywhere yet. On the contrary, I keep assuming you have them; perhaps I should stop, as you ignore what I write, and your only arguments are "but CPI=BPP, and even krugman says so", completely disregarding everything I write. Is my explanation that CPI can only equal BPP if they were constructed to be so wrong in any way?
> but you are the one who used the anecdote of your personal purchases as evidence.
No, I brought this up as anecdote. I did bring independent evidence (from a seekingalpha author) that agrees with shadowstats analysis - which you conveniently ignore.
> For every case like this, there are 100 where the professional consensus was right, and the lone dissenter was simply a quack.
This is a meaningless statement of appeal to authority. I'm not debating thousands of cases. I'm debating one case, with some support, which you conveniently ignore.
> You guys haven't proven beyond a doub
Who are "you guys?"
> You are also accusing BPP of fudging their numbers, and admittedly they aren't open about their data sources, but here's the rub.
Sorry sir, you apparently lack both statistical skills and reading skills. I am not accusing BPP of anything of the sort. I would urge you to reread what I wrote, but I have done that already, and that is useless.
For the benefit of others who actually read (if anyone still is) - the BPP index cannot have been constructed other than to have a value similar to the CPI. They do not have to be fudging or part of a conspiracy to be wrong (a fact jshen conveniently ignores)
> You could easily create your own BPP type index, make it open with the code in the public domain, and show some real numbers to prove that the BPP is fudged. Right?
"Easily?" Are you aware that BPP is the only (semi-)independent source of this data, and when they were about to close down there was a lot of outrage at the lack of any alternative.
In fact, there are millions of dollars to be made in knowing the difference between the officially published CPI and the "analyst estimate consensus". If you think it's easy, do that and you can retire on the first time your CPI collection disagrees with estimate consensus. Similarly for NFP numbers, PPI and others.
"Easily?" - one of us is living in lala-land, and it isn't me.
> side note: shadowstats is not transparent either. They haven't produced the data sources plus the code that generates their numbers.
Neither does the BLS, by the way, only very vague descriptions, and parts of the data are based on interviews asking people to speculate "if you were to do this, how much do you think you could earn / how much would it cost".
There is NO WAY for anyone to independently verify significant components of the BLS' CPI number.
Why is it so hard to create an index of online prices, which is what BPP is.
Here is their own explanation:
"Data collection: our data are collected every day from online retailers using a software that scans the underlying code in public webpages and stores the relevant price information in a database. The resulting dataset contains daily prices on the full array of products sold by these retailers. Our data include information on product descriptions, package sizes, brands, special characteristics (e.g. “organic”), and whether the item is on sale or price control."
That isn't hard to do in my mind. Why do you believe it is so hard?
Let's be clear. You are claiming that inflation is significantly higher than the CPI or BPP. You can talk until your blue in the face about the "problems" with those two, but you have to also provide some alternative data/math which is transparent. shadowstats is not transparent. It seems rather easy to me to make a price index, make the data public, and make the code public. I'm happy to help out on the coding side if you're up for it.
This is my last reply in this thread, because you are either trolling or have no idea what you are talking about, and I'm just wasting my time.
> Why is it so hard to create an index of online prices, which is what BPP is.
Why is it so hard to create a searchable index of the internet, which is all Google Search Engine is?
An index of online prices is much much smaller, say "just" 1% of the effort, so doing it dependably would cost "only" $5M/year or so to produce and maintain if you paid market prices (BPP has cheap student labor, I would guess, that makes their cost much lower; also, they can get away with not being robust). It's not just about crawling retailers - you have to be able to pull the prices and canonicalize products properly, and account for site layout changes. I actually did that in a former startup of mine. Doing it reliably is damn hard; embarrassingly parallel -- you could do it quickly by just throwing more money at it, but a ridiculously large amount of detail is involved.
Yes, yegg has been able to pull DDG with much, much less, but the talented people at Cuil were unable to pull with much, much more. If you're as good as yegg at this, I bet this is more rewarding (both financially and otherwise) than your job now. I know I can't pull a robust price index for less than a few $M.
> That isn't hard to do in my mind. Why do you believe it is so hard?
Because I live in the real world, and not in your mind. I actually did it in 1999-2001 for a startup. Technology has made it easier since, but not much easier. We developed a web scraper back then that was on par with everything available today (and is still ahead of many modern web automators like selenium).
Sites like pricegrabber are worth tens of millions if not hundreds of millions (meaning, if you were a company wanting to do that, you would either build-your-own or buy one for such a sum - so it's a good estimate of the costs, to within an order of magnitude), and all they do is create an index of online prices. In 2000, MySimon was sold for $700m for being nothing more than a price index. It wasn't a good deal for the buyer, but $70m today is probably a steal for a good, up-to-date price index of the web.
> That isn't hard to do in my mind.
I'm done here. I don't know what your real world experience is like, but it apparently doesn't help understand the involved statistics, economics, business and operational complexities. You demand details but you ignore them when given. I guess it's really a nice place in your mind.
> but you have to also provide some alternative data/math which is transparent.
BLS data is not transparent, and more than 30% of the value is officially based on speculation (a fact you conveniently ignore ignore). Ben Bernanke says it's accurate so it must be so. In the same way Europe is contained (as he said in 2008, 2009 and 2010), the mortgage crisis is contained (as he said in 2008 up until 2011), there is no chance whatsoever the US will be downgraded (as himself and Tim Geithner have repeatedly said when asked, up until the point it happened), etc.
If you actually read the data in http://www.bloomberg.com/news/2012-04-19/cpi-conspiracy-theo... you'd note that (a) there actually are no broad checks involved despite the title, and (b) while the BLS is not the only one using said methodologies, they are not universally accepted (not more than 40% of comparable countries use them) as it is claimed. Hedonics adjustments are more recent than the "recent" review of 1989 -- a review which was conducted with the implied target of finding that the BLS overstates consumer price inflation (by virtue of being commissioned by a congress quest to cut budget -- and what do you know, that's exactly what they found out!).
> It seems rather easy to me to make a price index, make the data public, and make the code public. I'm happy to help out on the coding side if you're up for it.
Since you live in your mind, and I live in the real world, it would be hard to bridge the gap and work as a team.
I've written a web crawler for a fortune 20 company, while I was in a research group at said company. I know exactly what would be involved. Storing prices with some small meta data is much much cheaper than storing full pages in the form of a search index. Scraping a discrete list of sites is far far easier and cheaper than writing a crawler robust enough to attempt to crawl arbitrary pages on the internet. Not only that, many retailers have APIs these days so a scraper wouldn't be needed for many of them, and reading from an API is something a novice programmer could do in less than an hour per retailer.
One of my close friends worked at pricegrabber for years, and the statement "all they do is create an index of online prices" is completely false.
Your continual use of ad hominem speaks volumes, and is your attempt to hide a lack of hard numbers.
> Your continual use of ad hominem speaks volumes, and is your attempt to hide a lack of hard numbers.
Questioning your reading skills you ignore given hard numbers that you've requested is perfectly reasonable. Questioning your math or economics skills when you insist that BPP and CPI are independent when you admit to not knowning how they are derived is perfectly reasonable. You might want to check what ad hominem actually means (beyond the latin translation), you might be enlightened. Alternatively, you can just ignore anything that contradicts your world view, as you have done before.
E.g., the modern definition of unemployment is indeed very different from the definition of 40 years ago, which would yield something like 20%-25% today (compared to 12% or so today). And yet, everyone assumes it is the same statistic. Same with inflation.
Personally, my 3 biggest expenses (rent, food, energy, in that order) have been going up at almost 10%/year over the last 3-4 years. I don't see a reason to doubt the shadowstats CPI metric.
You're welcome to trust BLS data. Just be aware that you can't trust them when they are published [ http://www.zerohedge.com/article/visualizing-propaganda-erro... ] and given that 90% of the values are revised to be worse within a month, their methodology cannot be considered statistically sound.
edit: added "almost" to 10%. Just checked, and rent was "only" 8%/year.