I don’t think the issue is how much of “science” is fake. It’s how you define “science”.
There have always been lower quality work and journals that are inherently given lower weight in the scientific community. Many of these lower quality papers just aren’t considered by working scientists. It’s a bit like email… once you start seeing a significant amount of spam, you just get used to ignoring it. This is a feature of the system. Scientific consensus can change over time, but it takes a lot of evidence to get there. But some papers get more weight than others.
For me, there are two real issues - first, public trust in science. When the public thinks that 14% or “science” is fake, they are likely to discount all scientific findings, regardless of source. This can be dangerous, especially for a public who is already distrustful of science/academia in general. So, instead of looking at results with a critical eye, the danger is that a large fraction of the public will just ignore all results.
The second issue is in polluting AI training. When you feed extra garbage into a system, you’ll get garbage out of the system. As LLMs get more specialized having high quality inputs for training is critically important. A contributing force here is that lower tier journals (those most susceptible to faked data), are also the most likely to make the full text of papers widely available. So, the easiest data to get for training is also the most likely to have faked data.
14% is a lower bound. I suspect if you include studies that can't be replicated it's well over 50%. And yes with that much garbage the public should dismiss all new published scientific findings. Modern scientific publishing has been gamed it's nearly useless for learning anything directly because you can't trust it. Some fields more than others of course, but it's all bad. Yes this includes tier 1 journals.
This has always been true? The vast majority of scurvy research was unreplicatable bullshit, and for hundreds of years if you “Trusted the Science(TM)” on scurvy, you were liable to die horribly on long ocean voyages. Mind you, if you didn’t Trust the Science you’d die just the same.
Then, someone worked out a theory and replicable experiment (humans and guinea pigs are the only mammals that die horribly when not fed vitamin C) and with a replicable experiment in hand, you could use little s science and now no one gets scurvy, trust not required.
Imagine someone is caught in a fraud, and a commenter responds "There are two real issues - first, public trust in [person]." Wouldn't that priority strike you as strange? Obviously distrust is reasonable and it's much more important for the person to really become more trustworthy and for others to accurately understand the situation. The importance of science does not diminish this.
It's not just lower quality journals. Marc Tessier-Lavigne was removed as the President of Stanford for having faked data in 12 of his papers, some of which were published in Science. He's still a professor though and still publishing. There are a whole bunch of Alzheimers studies that have been shown to have been fabricated, and led to extremely expensive trials on Alzheimers drugs that didn't work.
It also how you define "fake", eg in machine learning it can be everything from fabricated results to data leakage, low quality data, bugs, p-hacking, weak baseline design, insufficient handling of randomness, etc etc.
The argument that science should be trusted (believed) is insane, the whole point of science is questioning everything and deferring skepticism only in lieu of anything that can disprove.
Trusting science is just another form of religion, especially since science ("science"?) is often positioned as the rival to religion.
Nothing is _ever_ certain or completely specified in science (or elsewhere)... and society is going to move forward, so what level of 'evidence' is sufficient for you? This is going to come across as antagonistic, but...what's the alternative? "The argument that science should be trusted is insane..." ... so are we expected to twiddle our thumbs and do nothing for the rest of time?
I'm assuming you trust at some point? Have you ever flown? We have theory, we have evidence, and a high level of _justified true belief_ (to refer to recent HN posts) for how flight works, but perhaps all _these_ replications are just fortuitous and 'fake'? We 'trust' Newtonian physics enough for society to _act_ and _progress_ ... and then we quantum comes along.
>so what level of 'evidence' is sufficient for you?
Enough evidence such that no evidence to the contrary can stand.
We agree that liquid water is wet because we can't find anything (yet) that says liquid water is not wet, it's simple as that and science is ultimately as simple as that.
>Have you ever flown?
Yup, I'm a frequent flyer actually and fly more or less every month or two for business (and pleasure once the business is done, but that part's a secret).
>We agree that liquid water is wet because we can't find anything (yet) that says liquid water is not wet, it's simple as that and science is ultimately as simple as that.
I find this a weird example of something that has "Enough evidence such that no evidence to the contrary can stand."
"Water is wet" is a definition, not a scientific finding/description.
"Water tends to adhere to, and penetrate, materials." That's sort of a scientific finding.
"We call this condition wet" isn't so much.
Science, by Popper's definition, is something that can be falsified but has not yet been so. If no evidence could contradict our current findings then, by definition, our current findings are not falsifiable.
I realize that I'm quibbling about definitions, I guess I just want to point out that your level of evidence is dangerously close to being a self reinforcing tautology. Similar to how any religion justifies itself.
The point is that our agreement is merely the result of there being no other conclusion as far as we are aware, and to expand our awareness we must be skeptical of our conclusions in perpetuity.
This is why trusting science is insane, because if you trust something you stop questioning it and if you stop questioning then you never know if it's true or false.
>Science, by Popper's definition, is something that can be falsified but has not yet been so. If no evidence could contradict our current findings then, by definition, our current findings are not falsifiable.
I'm going to ignore that you just defined "current findings" as not science, since I presume that wasn't what you intended.
Just because I listen doesn't mean I'm going to become.
I like making my own opinions, because that's how I become informed. If I'm just taking in what others say, I'm just an NPC acting along their programming.
There have always been lower quality work and journals that are inherently given lower weight in the scientific community. Many of these lower quality papers just aren’t considered by working scientists. It’s a bit like email… once you start seeing a significant amount of spam, you just get used to ignoring it. This is a feature of the system. Scientific consensus can change over time, but it takes a lot of evidence to get there. But some papers get more weight than others.
For me, there are two real issues - first, public trust in science. When the public thinks that 14% or “science” is fake, they are likely to discount all scientific findings, regardless of source. This can be dangerous, especially for a public who is already distrustful of science/academia in general. So, instead of looking at results with a critical eye, the danger is that a large fraction of the public will just ignore all results.
The second issue is in polluting AI training. When you feed extra garbage into a system, you’ll get garbage out of the system. As LLMs get more specialized having high quality inputs for training is critically important. A contributing force here is that lower tier journals (those most susceptible to faked data), are also the most likely to make the full text of papers widely available. So, the easiest data to get for training is also the most likely to have faked data.