I think describing it as 'fake' is rather antagonistic. You could just as easily say 'poor quality' or 'unjustified'. I would say that more accurately describes the situation.
It could be...
- unjustified and sloppy (not 'fake', but also not considered reliable evidence)
- unjustified and malicious (this i would consider 'fake')
- unjustified and gamed (again, 'fake')
- ...or just unjustified and under-specified (and would result in 'true' results if the conditions for replication were better studied and defined)
Lots of people like to think themselves smart for following 'first principles'...and then often end up falling in the same ditches. First principles + received wisdom is a bit of a contradiction... if it's 'wisdom' rather than evidence, you are skipping your principles to go with the received starting point...
The problem is, that as soon as a paper is published in a peer reviewed journal, it can be used to justify public policy positions. Then a large number of people will points to the study, or even worse, to press characterizations of what a person thinks the study says, to support their preference on the public policy position. Immediately, anyone who has an alternate position on the public policy will be accused of being a “science denier.”
Almost no one is interested in having an honest discussion about whether or not the original paper actually says what it’s characterized to have said, and whether it was a good study in the first place.
So nowadays, when public policy is concerned, largely I disregard any scientific study that is introduced to support any position on the policy, and just do my own cost – benefit trade-off to determine my policy position.
This is a problem with the journalism and politics, it's not really about science. No scientist would trust a result that depends on a single small sample paper. Those are just stepping stones that may justify further research for more robust evidence. This fact is quite clear to scientist and it's why most would discourage the general public (including smart engineers) from reading academic articles.
But in general, I agree with you. It's ridiculous when someone pretends to shut down a complex issue by citing a random paper. However, an expert can still analyze the whole academic literature on a topic and determine what the scientific consensus is and how confident we are about it.
When I checked how people were citing these useless papers, almost invariably it would be in a sentence like this:
"Computational modelling is a useful technique for predicting the course of epidemics [1][2][3][4][5]"
The cited papers wouldn't actually support the statement because they'd all be unvalidated models, but citing documents that don't support the claim is super common and doesn't seem to bother anyone :( Having demonstrated a "consensus" that publishing unvalidated simulations is "useful", they would then go ahead and do another one, which would then be cited in the same way ad infinitum.
I disagree. A scientist could read a single paper and find out n is small, or identify a flaw.
But there are loads of papers like this.
Then you have some literature studies which look at all these papers together and get result aggregates.
Then you get some “proper” studies which link to these aggregates, and several small studies, and you’re going to read these “proper” studies which are quoted often and deemed decent or good quality.
And at no point will you realise it’s all based on shoddy foundations.
This is for example what recently happened in social psychology
You say you do a cost:benefit...but where do those costs come from? To me, that's just voluntarily doing your own ignorant, sloppy science I was mentioning above. If you only consider the blatantly obvious costs and benefits, you are completely ignorant of any 2nd or 3rd order...or even your own blind spots. You may radically under or overstate, or even calculate in the wrong direction.
I think a better position is that we should have a higher bar of what level of study or replication is required for a given situation...whether that be health, housing, infrastructure or whatever policy is coming in...what kind of monetary outlay and timeline of impact is expected. I don't think most people here would be happy with a 6-person study, unreplicated, deciding policy...so what IS the threshold?
I don’t analyze the papers, I analyze the policies.
Take the climate. Assuming the science is correct (storms worse, oceans rising, etc), let’s do a cost benefit analysis of the proposed policies.
The proposed policies in the US all dramatically increase the cost of energy (and therefore of everything), but only slightly slow the progression of warming, AND the bigger contributing countries in the developing world, esp China and India, will continue or increase their CO2 output.
We already know how to build dikes, and people who buy oceanfront property already know the risks, and over the last century we as a species have gotten really good as reducing deaths due to weather, so I don’t support the draconian carbon reduction proposals, and instead we can just deal with the side effects as they arise.
But how do you analyze the policies without doing science? Nothing in the above is sound to me.
"The proposed policies in the US all dramatically increase the cost of energy" - why? How do you even begin to conclude this without looking at some sort of (economic/scientific) analysis?
"only slightly slow the progression of warming" - again, how are you concluding this?
"we as a species have gotten really good as reducing deaths" - why should this trend continue? Why should it continue in the face of more extreme weather/climate change?
All I see are things you _think_ are true, and so to you your argument seems sound. But as the comment you replied to said, all I see is ignorant, sloppy science, since any meaningful analysis of these policies is by definition science. These cost/benefits you mention are not universal apparent truths.
I’m don’t need to analyze any scientific paper to form an opinion on a matter of policy; what scientific paper should I read to decide whether to support an increase on property tax in my county to support schools? The idea that I have to defer my judgment to someone in a lab coat is insulting.
As to your other questions, the answer to all (which any trivial web search would answer as I did) is “I can read”.
And I didn’t “cherry pick” some particular study or some blog that supports my preconceptions. All of the links I have provided are from sources that assume severe climate change is coming without serious interventions in greenhouse gas emissions; they are all on the “climate activist” side.
And of course now I’m being down voted for my example.
I never challenged a single point of climate orthodoxy; I never questioned whether global warming was occurring, whether it was man-made, whether the studies and models predict warming accurately, or whether the impact will be as stated. I stipulated every single one of those points. But simply my personal conclusion that I don’t believe the policy trade-off is worth it, means that my voice is not worth hearing.
From my perspective, I don't think I ever see "the science" being decided based on a single crap paper, so it seems a bit of a skeptical straw man. Perhaps that's different in your country, state, city, county, continent, island. Perhaps that's true for a particular policy _area_ that you are invested in. In general, I find arguments on scarequotes "science" often overblown and just reinforcing existing biases.
How much 'policy' are you happy with that you've never checked to validate that it was properly replicated beforehand? How much of your skepticism is against "the science" on new policy you start with an initial dislike to...compared to policy that we consider as standard, are probably comfortable with, likely consider to be 'beneficial enough'...and yet haven't adequately scrutinized?
And sometimes the opportunity window for studying a thing is just closed forever. Unique events are unique. That strange meteorite will not return until 300 years, or the species used in the experiment split in two, making impossible to duplicate it.
Fake is a good word. "unjustified and sloppy" is fake to me. We have different definitions of fake and shouldn't get hung up on definitions. My last word on it is I should be antagonistic against people who are deciding what goes in my son's school lunch based off of "unjustified and sloppy" studies.
First principles + received wisdom are counter balancing, not contradicting. Everything in life is a balance.
Definitions matter when it comes discussion, as what you say influences how people feel on a topic. Broad, non-specific definitions leave a lot of space for bias rather than clarity.
If you describe it as 'fake', I consider that to give the impression of 'the answer is NOT' this, and could lead to anti-policy.
If the description is 'unjustified and sloppy', that can lead to additional research to properly invalidate or potentially find something useful, so we don't throw the baby out with the bathwater.
It could be...
Lots of people like to think themselves smart for following 'first principles'...and then often end up falling in the same ditches. First principles + received wisdom is a bit of a contradiction... if it's 'wisdom' rather than evidence, you are skipping your principles to go with the received starting point...