Your comment is broadly misleading. In fact, I would say that "shadow stats" guys like you have enabled the destruction of the system by creating the space to cast doubt on the valid methods used by BLS. BLS unemployment metrics have a valid basis and where they differ from Eurostat those differences are minor and with rational basis (such as 16 vs. 15 year old starting age).
It is tough, though, for me to fully buy labor statistics when it has become the norm recently for them to be revised down. This spans back into Biden's term as well so it isn't one party either.
With a valid measure I would expect a roughly even distribution over time between underestimates and overestimates. For a valid measure worth considering I'd also expect the stat to be released later when revisions are less likely because more actual data has been collected
> With a valid measure I would expect a roughly even distribution over time between underestimates and overestimates
This is a valid hypothesis. It’s wrong, and I’ll explain why. (It’s a bad and invalid thing to conclude.)
If measurement errors were iid, you’d be correct. But they’re not. They’re well documented for not being so. Earlier survey results are biased by directional response bias inasmuch as the employers with the lease changes respond first. So the earliest releases tend to match whatever was going on before. Then the employers who had to do paperwork respond. And then, finally, someone gets around to calling the folks who never got back. Some of them aren’t around anymore.
So yeah, the directional tendency in revisions is well documented. And for a long time, the early releases were appreciated. But maybe American statistical and media literacy is such that only final releases should be released, which would mean we’d always be working with data 6 months to a year out of date.
That's all well and good in theory, but job reports data over recent years have noticeably shifted towards downward monthly revisions. Prior to the pandemic response, the graph [1] looks much more balanced with regards to positive and negative decisions.
Sure, but it's totally ridiculous to post about that without discussing the survey response rate, which is the cause of that drift. People are attributing it to political meddling, and that is baseless.
Naturally all of this metadata about the BLS surveys is available for free from the BLS, so you can just go look at it.
Interesting that you're claiming this is baseless without providing any sources for your alternative. How do you know that (a) the response rate is down meaningfully and (b) that data shows a strong correlation or causation between the two?
> but job reports data over recent years have noticeably shifted towards downward monthly revisions. Prior to the pandemic response, the graph [1] looks much more balanced with regards to positive and negative decisions
Yes. The reasons for this are well documented. Changing methodology for the preview estimates is rigorous. That means our published estimates lag best estimates, something the primary sources note in every release if one gets past the headlines.
Also, if you have one year of massive job gains and four years of flat and falling, you’ll spend most of your epoch biased one way. Again, not a sign of methodological problems. Just a predictable methodological artifact that folks are supposed to be able to incorporate before using, much less emotionally reacting to, the data.
Why would the shift to a new methodology bias the estimates to one end? I would expect a new methodology to make comparisons of data between the two systems to potentially be unhelpful, but I wouldn't expect a valid methodology to bias one way or another.
Related, I wouldn't expect past data to bias a current estimate. If 6 or 12 months of positive growth biases the next prediction it falls into the hot hands fallacy. It isn't predicting based on current predictions, its predicting based on recent past behavior and extrapolating forward. This only makes sense to do if the data is not yet available, and even then the extrapolation isn't a useful estimate of current conditions.
> If 6 or 12 months of positive growth biases the next prediction it falls into the hot hands fallacy
It’s a sample of a sample. The full sample is the final release. The early results are the preliminary releases. When firms change things they take longer to respond. So whichever way the economy is moving, there will be bias in that direction. If the economy is turning, you won’t know direction. If it’s accelerating or slowing down you don’t know magnitude. Sometimes context clues can help. Sometimes they can’t. There is no known statistical treatment for intuiting the missing data before one has it11
We agree here, and I am going a step further saying that the initial numbers are useless and are little more than throwing opinionated darts. Numbers shouldn't be released until they meet some reasonable level of response and statistical validity. Given that they do release numbers today, I judge them as early and either inaccurate and useless or politically motivated to push markets while there's no meaningful data to contradict them.
> the initial numbers are useless and are little more than throwing opinionated darts
You’re still concluding from ignorance. They are not. A better question would be ask to whom they’re useful and how.
Like, if a fire is burning in a neighborhood, every sighting is valuable. You don’t always need to wait for a comprehensive picture before being able to do anything.
> I judge them as early and either inaccurate and useless or politically motivated to push markets while there's no meaningful data to contradict them
That’s wrong. But it seems to be a common error.
Maybe the solution is to make these numbers available only to gatekeep these numbers. Policymakers, academics, enterprises and banks can get a rarefied sheet for a fee. But the public doesn’t get PDFs, much less public reporting.
> while there's no meaningful data to contradict them
There are bajillions of them. ADP. State reports. Private surveys. Fed studies. That said, I’m leaning towards your view—maybe these data aren’t best made broadly public.
> Like, if a fire is burning in a neighborhood, every sighting is valuable. You don’t always need to wait for a comprehensive picture before being able to do anything.
That assumes there are a meaningful number of reliable reports. If I regularly am told there is a fire only to have authorities come back a week later to adjust reports down I wouldn't trust them. If they over estimated the number of fires based on the last 6-12 months of fire data, with little recent data to go on otherwise, I would ignore the reports.
> Maybe the solution is to make these numbers available only to gatekeep these numbers.
This seems more reasonable at least, though I don't see much use in the data still when its released so early that its based primarily on recent historic trends and few survey responses.
> There are bajillions of them.
Those are all anecdotal in this case. If said sources were applicable and reliable the official data would consider those and have more accurate reporting. My point was that said reports depend on survey results, and when they report early results so early that few responses are in yet them there is no official data to contradict the early reporting.
That is a reasonable position, however the assumption that it is the administration that is gaming them vs other motivated parties is open for discussion.
It is in fact not at all reasonable. They are saying that the BLS stats can't be trusted because they totally misunderstand the survey methodology. That isn't a reason!
I’d counter that if we were doing a good job gathering data that these structural biases could be compensated for with more conservative initial numbers.
At some point a lack of decision to take compensating action becomes faking the numbers.
> if we were doing a good job gathering data that these structural biases could be compensated for with more conservative initial numbers
There is no more conservative. The data will bias in the direction of trend. The point of the data are, in part, to measure that trend. Fucking with it to make it politically correct to the statistically illiterate is precisely the sort of degradation of data we’re worried about.
(They’re also useless as a time series if the methodology changes quarter to quarter. That’s the job of analysis. Not the data.)
What you wrote suggests the data will bias predictably, which matches my understanding.
Reporting biased data as the default because the bias compensation is already built into the audience seems like a weak argument for not improving.
They can provide for the continuation of data visibility/granularity by releasing the prior numbers as previously calculated and at the same time changing the calculation of the headline number to be better compensated.
The simpler argument is that changing it at all will result in a negative step change in the reporting that no one wants to take accountability for.
> What you wrote suggests the data will bias predictably
Ex post facto. Before the fact, we don’t know.
Imagine you know the weather will be a strong gust regardless of direction. Averaging the models will produce a central estimate. But you know it will be biased away from the center. You just don’t know, until it happens, in which direction.
> They can provide for the continuation of data visibility/granularity by releasing the prior numbers as previously calculated and at the same time changing the calculation of the headline number to be better compensated
They do. These data are all recalculated with each methodological change. They’re just deprecated indices the media don’t report on because they’re of academic, not broad, concern.
> simpler argument is that changing it at all will result in a negative step change in the reporting
Simpler but wrong. Those data would be useless for the same reason we don’t let CEOs smooth revenues.
I’m confused by this discussion. It seems like you said the biases were structural because we know who reports early and that is why the early numbers are always revised down. Structural implies known in advance.
It also seems like you said they shouldn’t revise the numbers but now you are saying they already do.
> It is tough, though, for me to fully buy labor statistics when it has become the norm recently for them to be revised down.
There have been revisions since the forever, and this is because they depend in part of surveys, and if companies (and the people with-in them) don't bother responding in a timely or accurate manner then that's going to throw the sampling off.
> CES estimates are considered preliminary when first published each month because not all respondents report their payroll data by the initial release of employment, hours, and earnings. BLS continues to collect payroll data and revises estimates twice before the annual benchmark update (see benchmark revisions section below).
Post-COVID surveying seems to have become more difficult (and BLS budget stagnation/cuts haven't helped). This has been a known issue for a while; see Odd Lots episode "Some of America's Most Important Economic Data Is Decaying":
> Gathering official economic data is a huge process in the best of times. But a bunch of different things have now combined to make that process even harder. People aren't responding to surveys like they used to. Survey responses have also become a lot more divided along political lines. And at the same time, the Trump administration wants to cut back on government spending, and the worry is that fewer official resources will make tracking the US economy even harder for statistical departments that were already stretched. Bill Beach was commissioner of labor statistics and head of the US Bureau of Labor Statistics during Trump's first presidency and also during President Biden's. On this episode, we talk to him about the importance of official data and why the rails for economic data are deteriorating so quickly.
My argument wasn't that there shouldn't be revisions though, only that recent years have shown consistent negative revisions rather then a roughly even distribution.
If response rates are down or something else is making surveys more difficult, its reasonable that confidence windows would weaken and size of revisions would increase. Its unreasonable that difficulty in surveying would lead to a consistent bias in results though, that's a methodological issue at best.
> My argument wasn't that there shouldn't be revisions though, only that recent years have shown consistent negative revisions rather then a roughly even distribution.
It's been to too many moons since I took a prob/stats course to comment accurately on population sampling, but how valid is the assumption that errors 'should' skew both positive and negative?
If errors are skewed in one direction there would likely have to be a factor forcing it, like sampling and response bias.
That's always possible, though again I question the validity of the measure and results if its getting consistently skewed results. Either the methodology is faulty or the results simply can't be trusted because they can't reliably get good data.
I don't say stuff like this very often, but are you actually blaming a victim for dealing with the reality of government bsing its own stats instead of the government that allowed this bs to continue? BLS had only one thing going for it and it is mostly that it was used for long enough time that changing methodology would prevent us from being able to compare it prior time ranges. That is it. Otherwise, the methodology itself is seriously flawed ( and likely was from get go, but these days, it is absolutely the worst possible mix of options ).
Honestly, your comment made me mildly angry. That said, can you say why you believe parent's comment is misleading?
I've never met a single person willing to attest to filling out a BLS survey. Not once. If their methodology is built on that + unemployment data from State Unemployment agencies + data from payroll processors, anyone not collecting state unemployment benefits is invisible to the system, and half of the payroll is actually not even consituted of U.S. Citizens.
Admittedly, if I could find a single instance of someone willing to vouch or share insight on having filled out a BLS survey, that'd cure a healthy chunk of skepticism. There's still be the other distortions in the data to account for, but I'd at least have an instance proving that yeah, there is somebody filling out these surveys and it isn't just something they say they do to make their magic unemployment number sound legit.
Note, I'm in a massive sceptical shit phase at the moment. Last decade has burned my optimism hard. So when it comes to my ability to assume benevolent intent right now, there's a heavy bias against doing it, and a heavier bias in the direction of "what would be the easiest way to keep the System limping along?" The answer to that is "say you do one thing, in reality do another, and as long as no one comes lookin', it's gold." The finance industry runs on Trust moreso than anything else, and there ain't much to be said for Trusting anything you can't verify these days. Not from other humans.
> if I could find a single instance of someone willing to vouch or share insight on having filled out a BLS survey, that'd cure a healthy chunk of skepticism
See, Census letters are one thing. BLS is another. I've actually received Census letters. BLS ones, not so much, and given they claim to be collecting data through surveys all the damn time, I'd expect to have been able to find someone who filled one out. It's weird, to me, that my luck has been so bad in finding someone with context on it. At best I only find someone who knows BLS uses surveys as part of their methodology, usually through reference to the site. No one ever seems to be able to primary vouch for having been the one surveyed.
>How might one distinguish such "scepticism" from ignorance?
Ignorance doesn't seek to invalidate itself. Scepticism does. It does not enrich my life knowing there's a methodology to collect a "high value statistic", but not finding any on the ground proof of people who actually have primary exposure to the methodology. One can't reason around what the system is actually measuring without a sampling of that. I can find screenshots of UI at times. I find papers around low response rates. I never find an actual person who says "Yeah, I get dinged to do those every few months, once every few years..." I sure as hell know when I'm gathering statistical data, that sampling bias evaluation requires foot work, and if you do that footwork, if your methodology works, it shouldn't take you that long to run into someone you've surveyed if you're doing it right. If you're not, and only hitting "the usual suspects", you're not getting a representative sample/measuring what you think you are. So I look for payroll people or people who have done payroll in the U.S. and ask if they've actually ever been directed to provide input. I've been doing it the last few years. Nobody seems to recall ever having been asked to participate in what amounts to billion dollar money movement at stake jury duty.
So yeah. This is kind of a weird tic of mine at the moment. Ranks right up there with the time I felt the inexplicable urge to figure out what the deal with zoning as applied to city planning was and how it worked. Something just doesn't add up. I hate that. Mental equivalent of a thumb detection via hammer.
> I've never met a single person willing to attest to filling out a BLS survey.
Unless you have introduced yourself with this question to thousands of people, this is a totally meaningless statement. It says more about your social circle, your grasp of descriptive statistics, and the weird online stew you are soaking your brain in than it says about the CPS.
I never complained about the Current Population Survey. Census Bureau does a great job. I complain about BLS. And yes, I'm the odd fellow who does, in fact, talk about weird niche 'work' minutiae with people because I'm actually curious how the world works, and whether our "authoritative metrics" are actually worth a damn. My grasp of descriptive statistics is just fine, thank you very much, but I also happen to know it doesn't work if you're not actually collecting data, and if the data you're collecting is incomplete, you're doing what is essentially inferential statistics (assuming your whole is entirely represented by what you have collected. If you don't check, you don't bloody know. If you actually are checking, it's not normally that hard to find the population you're checking with.
I can't tell if you are serious or not. Lets assume for a moment that there was once a benefit to BLS survey methodology ( I would argue otherwise, but w/e ). Is it a good methodology today?
So my main argument ( and frankly the only argument that should matter ) is that is a bad fit for the goal of estimating values ( even though we do know its failure modes ). Is that not enough?
You made the argument and provided zero supporting evidence. As it stands, it's merely an opinion, and appears to be an uninformed one until you prove otherwise. That's what people are asking you to do.
Sigh, your supporting evidence is a record of someone saying something, which itself is merely an opinion.. men in glass houses and all that. The interesting thing about my opinion is that while it may not be AS informed as yours, it is notably above the average level of knowledge when it comes to BLS.
<< That's what people are asking you to do.
No. What I am being asked to do is: "Show me a better way, but I only accept a better way that is already utilized by someone else". Not a recipe for a thoughtful exchange of ideas.
Alternative is to build something better. Just about anything is better than the current survey system. What I would propose is something akin to "derived real-data unemployment system". All this data exists now, but is distributed. It can be stitched together, but if one was so inclined.
