> Why so angry?
I know I’ve taken this far too personally. I have no illusions that everything I read online should be correct, or about people’s susceptibility to a strong rhetoric cleverly bashing conventional science, even in great communities such as HN. But frankly, for the last few years, the world seems to be accelerating the rate at which it’s going crazy, and it feels to me a lot of that is related to people’s distrust in science (and statistics in particular). Something about the way the author conveniently swapped “purely random” with “null hypothesis” (when it’s inappropriate!) and happily went on to call the authors “unskilled and unaware of it”, and about the ease with which people jumped on to the “lies, damned lies, statistics” wagon but were very stubborn about getting off, got to me. Deeply. I couldn’t let this go.
I am afraid I actually agree with the author's point. The anti-intellectual, anti-scientific streak in many poor analyses claiming to debunk some scientific research is deeply concerning in our society. If someone is trying to debunk some scientific research, at least he should learn some basic analytic tools. This observation is independent of whether the original DK paper could have been better.
That said, I give the benefit of doubt to the author of "The DK Effect is Autocorrelation." It is a human error to be overly zealous in some opinions without thinking it through.
Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs.
There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand mathematical statistics and the philosophy of statistics fairly deeply. There just isn't any way around it.
I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. That is a problem and it does erode trust. But in some ways it's also how the sausage gets made, there's going to be room for things to get published they later gets refuted. People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becoming an expert yourself it's very difficult to know who or what to trust.
On the other end there's distrust of broad scientific consensus across different professions, countries, etc. It's the distrust at these levels that is the increasing problem we are facing today.
> I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published.
It's more than that, I think. Sibling-thread poster hit the nail on the head when he complained of politicised science.
The social sciences have this dominating and silencing effect on the rest of the sciences.
There's always been junk science, and when found out it gets discredited. This is still happening and is a good thing.
What's new is that any research that might produce results counter to the what the PC-mob deems acceptable is attacked. Whether or not there is consensus amongst researchers in that field is irrelevant when the mob calls for the firing of any researcher who doesn't toe the current political party-line.
Sure, we're not actually in the dark ages, but a trend of silencing voices in the name of purity of thought is particularly troubling, especially as the mob asking for this is unashamedly attempting to implement NewsSpeak[1].
[1] See the argument in yesterdays threads about what "man" and "woman" mean, and should dictionaries be changed, etc.
No this is not new. You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superior. There is still "science" being done to bolster those political views.
>> What's new is that any research that might produce results counter to the what the PC-mob deems acceptable is attacked.
> No this is not new.
I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently.
> You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superior. There is still "science" being done to bolster those political views.
I don't see what that has to do with that I said - that a very vocal bunch of non-science people seem to have successfully lobbied into silencing specific topics.
> I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently.
Are you genuinely serious or were you completely unaware of anti-communist government sanctioned blacklisting of academics suspected of being communist for political clout? Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Are you unaware of our own Alan Turing, of the Turing award, literally castrated not for his research but because he was a known gay researcher? Are you unaware of why HBSUs exist(black scholars were segregated for being black, their research dismissed because of the race of the researcher)?
>Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun?
Your point is good but this is a pet peeve of mine. Galileo was not punished by the church for saying the earth orbits the sun. Galileo was indicted and punished by the church because he was a local elite with several personal and political enemies within the church, and more directly, because he slighted the pope, his former friend, by taking a philosophical argument made by said pope, and putting it, paraphrased, into the mouth of a character in his book who was named "simplicio" and cast as a moron. That pope literally gave him permission to publish his claim that the earth orbited the sun, a claim which Galileo did not make based on science, but instead made because he felt the resulting (incorrect and based on outdated observations) mathematical model for the orbits of planets was more "elegant".
If the church truly wanted to punish him, they would not have sentenced him to literally stay at home in a beautiful villa and write books all day. His official charge was that lay people are not allowed to interpret the scripture, which he did a bit in his book. The church did not care if you made mathematical or scientific arguments about how the world worked. They only cared that you leave theology to the priests.
"Galileo was not punished by the church for saying the earth orbits the sun. Galileo was indicted and punished by the church because he was a local elite with several personal and political enemies within the church, and more directly, because he slighted the pope, his former friend, by taking a philosophical argument made by said pope, and putting it, paraphrased, into the mouth of a character in his book who was named "simplicio" and cast as a moron."
This is not really true. While there is some truth to the claim that Galileo placed an argument made by Urban VIII in the mouth of Simplicio and that Urban took offense at this, the trial documents, especially Galileo's sentence, make it very clear that Galileo was being punished for heresy and the heresy he was being punished for was the notion that Sun did not move and that the Earth did. From the sentence:
> We say, pronounce, sentence, and declare that you, the abovementioned Galileo, because of the things deduced in the trial and confessed by you as above, have rendered yourself according to this Holy Office vehemently suspected of heresy, namely of having held and believed a doctrine which is false and contrary to the divine and Holy Scripture: that the sun is the center of the world and does not moved from east to west, and the earth moved and is not the center of the world, and that one may hold and defend has probable an opinion after it has been declared and defined contrary to Holy Scripture.
Note that the term "vehemently suspect" is technical term. The Roman Inquisition in the 17th century didn't generally deliver straight up and down guilty or not-guilty verdicts and rather organized convictions according to degrees of suspicion. "Vehement suspicion" indicated that there was at least some (but not much) degree of plausible deniability that Galileo didn't believe what he had written, and that was only because Galileo denied it to the court.
"That pope literally gave him permission to publish his claim that the earth orbited the sun, a claim which Galileo did not make based on science, but instead made because he felt the resulting (incorrect and based on outdated observations) mathematical model for the orbits of planets was more "elegant"."
No, Galileo was given permission to publish a book that presented a neutral comparison of the Copernican and Ptolemaic models on mathematical grounds with the intention of proving that the Church was justified in its suppression of Copernicanism. Galileo's book was not neutral--it argued heavily in favor of Copernicanism--and that's why he got in to trouble. Urban VIII had been his friend prior to this episode so it's likely that had Galileo not placed Urban's argument in Simplicio's mouth at the end that Urban would have protected him rather than punished him, but the reason that Simplicio was given that argument was because that argument was intended to be the end of the book. After four days of continuously losing the debate, Simplicio finally raises Urban's argument about the omnipotence of God and his opponents are forced to agree with him. The idea being that Galileo could stick to the letter of his remit, while still arguing what he wanted. Unfortunately he argued too well and readers realized where is real sympathies lay. Urban VIII was accused of protecting heretics (not just Galileo, but also but supporting the French against the Hapsburgs in the 30 years war,) and so he made an example of Galileo.
Galileo's arguments were not simply mathematical. In fact, if they were, he would never have been punished because it was already permissible to treat Copernicanism as a purely mathematical hypothesis; the 1616 prohibition of Copernicanism explicitly carved out that exception. But Galileo used a wide variety of arguments, including physical arguments. Galileo after all, was primarily what we would call a physicist rather than an astronomer. It was Galileo's insistence that Copernicanism must be physically true and not just a better mathematical model that made him a heretic in the eyes of the Sacred Congregation.
It's important to note that the quality of Galileo's scientific arguments were never a subject of his trial. It was only his conclusions that the Sacred Congregation took issue with. Galileo's chief (but not only) argument, from the tides is now considered to have been spectacularly wrong, but at no point did that come up in the trial. Galileo's arguments could have been 100% perfect and unassailable (and scientific arguments rarely are) and he would have still been punished.
"If the church truly wanted to punish him, they would not have sentenced him to literally stay at home in a beautiful villa and write books all day."
He was sentenced to life imprisonment. That was commuted to house arrest on account of his old age and not at his own house at first. He was prohibited from receiving medical attention late in life. All books by him were placed on the index and he was prohibited from taking visitors. He did manage to continue his work, but that was by publishing in the Netherlands which was a Protestant country and hence outside the reach of the Church. He had first tried to publish in Venice which was a hotbed of anti-clericalism and usually Inquisitorial orders, but even they would publish him.
"His official charge was that lay people are not allowed to interpret the scripture, which he did a bit in his book."
That was not the official charge. I quoted the official charge above. Galileo's interpretation of scripture happened much earlier and preceded the 1616 prohibition of Copernicanism. There Galileo gave counter argument as to why Copernicanism didn't contradict scripture and that served as the catalyst for the investigation that led to the prohibition. Galileo ultimately was not censured for writing on scripture and his argument was even well received to a degree (he had checked it with a cardinal before publishing it,) but the Sacred Congregation decided that it was more concerned about undermining the authority of the Church Fathers, many of whom took the famous passage from Joshua literally, than it was about accidentally hooking scripture to a provably false view of the world. If you read Cardinal Bellarmine's response to Foscarini regarding Galileo's letter, you'll see him very clearly cite the authority of the Church Fathers as his primary consideration. Bellarmine controlled the Sacred Congregation at the time so his opinion on the matter was the Church's opinion.
"The church did not care if you made mathematical or scientific arguments about how the world worked.T hey only cared that you leave theology to the priests. "
The decree from the Index of Forbidden Books banning Copernicanism:
> This Holy Congregagtion has also leaned about the spreading and acceptance many of the false Pythagorean doctrine, altogether contrary to Holy Scripture, that the moves and the sun is motionless, which is also taught by Nicolaus Copernicus' On the Revolution of the Heavenly Spheres and by Dieage de Zuniga's On Job. This may be seen from a certain letter published by a certain Carmelite Father, whose title is Letter of the Reverend Father Paolo Antonio Foscarini, on the Pythagorean and Copernican opinion of the Earth's Motion and the Sun's Rest and on the new Pythagorean World System in which the said Father tries to show that the abovementioned doctrine of the sun's rest at the center of the world and of the earth's motion is consonant with the truth and does not contradict Holy Scripture. Therefore, in order that this opinion may not advance any further to the prejudice of the Catholic truth, the Congregation had decided that the books by Nicolaus Copernicus and by Diego de Zuniga be suspended until corrected; but of the Carmelite Father Paolo Antonio Foscarini be completely prohibited and condemned; and that all other books which teach the same be likewise prohibited, according to whether the present decree it prohibits, condemns and suspends them respectively.
Note that Diego de Zuniga and Paolo Foscarini are both priests. This wasn't about keeping theology to the priests, it was about prohibiting certain theology that would undermine the authority of the Church. The correction applied to Copernicus's book is that it be changed to suggest that his system was not intended as a literal interpretation but only as a mathematical model. As I mentioned earlier, an allowance for treating with Copernicanism as a pure mathematical contrivance for the convenience of astronomers was made but treating it as literally true was declared "error", and later upgraded to "heresy" during Galileo's 1633 trial.
Sorry for the essay, but this subject is a pet-peeve of mine.
Some good books on the subject:
1. Behind the Scenes at Galileo's Trial - Richard J, Blackwell
2. The Essential Galileo - Maurice A. Finocchiaro
3. Galileo Heretic - Pietro Redondi
Yes, I've TOF's blog posts before. TOF's perspective is one that I've encounter many times before. You mostly find it in some very conservative Catholic circles.
If I was being generous, I would say that TOF was pushing back against the overwrought hagiography that often surrounds Galileo and his confrontation with the Church. It's true that the common story is an over simplified account and there are some persistent myths that have worked their way into the story over the years into in order to make Galileo look more heroic and the Church more villainous than they either actually was. The real story is a bit complicated.
But the Church really did ban heliocentrism and it really did punish Galileo for arguing for it. That's not in question. I feel that TOF's argument otherwise rests on a deliberate misrepresentation.
TOF's argument the Church was actually just smartly waiting for proof before it changed it's doctrines is a bit rich in that Galileo offered proof and was punished for it. Now, Galileo's proof was bad, but that's not why he was punished. If it was, the Inquisitors would have mentioned that and not simply accused him of heresy for arguing that the Sun stood still. Not to mention that this whole idea rests on the bizarre notion that science works best when you silence debate until proof can be provided
Ironically, that whole argument stems from a quote from Robert Bellarmine's letter to Foscarini, where he admits that Galileo has a point about not interpreting scripture in a way that is provably false. In that quote, Bellarmine isn't saying that the Church is waiting for proof, he's saying that while proof would force him to change his mind, he doesn't think such proof is possible so he may as well go ahead and ban Copenicanism anyway. It's real obvious if you read the very next sentence that TOF for some reason doesn't quote:
> I add that the one who wrote, "The sun also ariseth, and the sun goeth down, and hasteth to his place where he arose," was Solomon. who not only spoke inspired by God, but was a man above all others wise and learned in the human sciences and in the knowledge of created things; he received all this wisdom from God; therefore it is not likely that he was affirming something that was contrary to truth already demonstrated or capable of being demonstrated.
In other words, he doesn't believe that Galileo or anybody else will be able to find proof that the sun stays still because that would contradict what he already believes based on his reading of the Bible!
So no, TOF is misrepresenting the stance of the Church in 1616. A lot of people repeat this because it looks like a clever debunking of a common myth, but it's actually more of bunking in that he inserts a lot of detail in order to disguise some blatant misrepresentation.
> Are you genuinely serious or were you completely unaware of anti-communist government sanctioned blacklisting of academics suspected of being communist for political clout? Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Are you unaware of our own Alan Turing, of the Turing award, literally castrated not for his research but because he was a known gay researcher? Are you unaware of why HBSUs exist(black scholars were segregated for being black, their research dismissed because of the race of the researcher)?
Every single one of those was NOT a mob.
The people in authority, using their authority to push their PoV, is very different to people with no authority forming a mob and demanding that the current authority silence other people from speaking their minds is a very different thing.
Whatever your view of the current authority is, it is infinitely better than mob-justice.
If you think castrating gay men wasn’t mob Justice at the time maybe you should reconsider. Same for whether or not black scholars were considered equal to white ones.
The real problem is that all of those mobs have to answer to the PC mob now. Or at least that's what I hear from people who think that censorship didn't exist until coincidentally around the time of Gamergate.
Mobs that try to force their own views on others are common. PTAs, or strong willed individuals with kids in school, are mob experts. Good things have come out of those mobs. Amsterdams good bicycle infrastructure was built on mob rule instead of listening to "rational engineers" who wanted to build motorways through the city.
Good science defines its terms. Can you unroll "PC-mob" so we're all on the same page here? You don't sound like an asshole, so your meaning is probably not the usual "anything that leans a little left."
Group think exists in science too, I mean Newton calling Leibniz a copy cat was a set back and it's crazy that I still had to learn about the priority controversy almost 300 years later. We feed on unneccessary controversy.
> People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becoming an expert yourself it's very difficult to know who or what to trust.
A problem here is that there are fields of science that are almost certainly bogus in themselves. One very likely candidate is nutrition, which seems to be fumbling in the dark and has a long history of producing worse recommendations than doing nothing (e.g. replace fat with sugar). More controversially, the entire field of economics is seen by some to be very suspect from a basic foundations view.
>A problem here is that there are fields of science that are almost certainly bogus in themselves. One very likely candidate is nutrition, which seems to be fumbling in the dark and has a long history of producing worse recommendations than doing nothing (e.g. replace fat with sugar).
It's not the field that is bogus in that case. People were quite literally bribed to push this. This could happen anywhere, anytime, in any field.
That was just an example, but we can go into more detail - at every point, nutrition studies are highly questionable. Sample sizes are usually minuscule (you can find important studies with 5-10 subjects, and even the largest studies rarely have more than a few hundred), they are often biased samples (only overweight people, only people with heart disease or diabetes etc.), they often don't account for likely confounding factors (using weight without accounting for muscle mass, no accounting for stress, time in the sun etc.) and on and on. And all of this in a subject where we basically don't have any clear idea about how much metabolism differs from one person to the next, based on what factors (e.g. gut microbiome has only been recently identified as a major component of digestion that can differ significantly between people; psychological effects of diet are even less well understood, even though your food choices are obviously not coming from some pure realm of reason).
Basically the digestive system is far too complex for us to understand from first principles at this time. Your diet has a very complex and very slow effect on your body, with some exceptions. Numerous diseases and environmental factors impact how this plays out exactly. So, to do real research in nutrition, the only chance right now would be to conduct massive studies over long periods of time with rigorous controls on subjects' nutrition and activities - which is basically impossible, or at least prohibitively expensive.
Instead, we get conclusions drawn from studies of a few dozen people over a few months or at best a year (in "long-term" studies). Or, we get conclusions drawn from comparing diet across huge populations ("the Mediterranean diet", "the Japanese diet", "the American diet" etc) with no possibility to control for obvious differences in nutrients, environmental factors, lifestyle differences, access to healthcare etc. Both of these are worthless conclusions, they don't tell you anything at all.
The only successes of nutrition science have been identifying the most basic nutrients we need to survive at a basic level (protein, fat, carbohydrates, and the various vitamins and minerals). Basically nothing beyond that should be trusted.
As a fun historical note, after the discovery of the macro-nutrients there was a budding field of nutrition scientists confidently recommending optimal diets using scientific methods. Unfortunately, they had no idea about the existence of micro-nutrients, so actually following some of their diets you could actually end up getting scurvy or other serious malnutrition diseases. The current slew is not that bad, but I wouldn't be surprised if in the future we will look back similarly at some common diet advice of today.
I think commercial operations like Huel and similar supplements will definitely be thought of like that. It just seems crazy, having watched the evolution of "scientific" diet advice over the past 70 years, to now think it's plausible that industrially-produced protein drinks are a suitable food substitute.
So, the reasons people learn a generalized distrust of science are that often the sausage doesn't get made. Bad science is published, applauded, cited, breathlessly covered in the media and may even be replicated, yet the first time outsiders to the field actually read the paper they realize it's nonsensical. But then they realize nobody cares because careers were made through this stuff, so why would anyone inside the field want to unmake them?
The degrading trust doesn't come from bad results per se, but rather the frequent lack of any followup combined with the lack of any institutional mechanisms to detect these problems in the first place beyond peer review, which is presented as a gold standard but is in no way adequate as such.
For example, consider how programmers use peer review. We use it, and we use lots of other tools too because peer review is hardly enough on its own to ensure quality. Now imagine you stumbled into a software company that held a cast-iron policy that because patches get reviewed by coworkers you simply don't need a test suite, nor manual testing, nor a bug tracker, code comments, security processes, strong typing, etc. And their promotion process is simply to make a ranking of developers by commit count and promote the top 10% every quarter, and fire the bottom 10%. Moreover they thought you were nuts for suggesting that there was any problem with this. You'd probably want to get out of there pretty fast, but, that's pretty much how (academic) science operates. So of course this degrades trust.
Maybe I'm missing something, but 'self-correcting' doesn't necessarily mean 'immediately self-correcting'. I guess it's safe to assume, that incorrect studies are not cementing our world view and entirely stopping us from questioning studied topics again.
The way I see the self-correcting nature of science: the truthiness of our view about specific set of topics increases over time (in some approximation).
Self-correcting doesn't mean immediately self correcting, but it does imply self-correcting in a somewhat reasonable time period, and ideally not needing to self correct too often.
What's reasonable, well, probably not years or decades. Average people cannot make major errors that destroy the value of their job output and then blow it off with "well but the company self corrected eventually so please don't fire me". When they judge science, they will judge it by the standards they are themselves held to in normal jobs.
And what's too often, well, probably papers that don't replicate should be a clear minority instead of (in some fields) the majority. Recall that failure to replicate is only one of many things that can go wrong with a study. Even if the replication rate was 100% many fields would still be filled with unusable papers.
Exactly this. It's even worse, distrust of broad scientific consensus is purposefully cultivated to further political and economic goals, and the methods to do so increasingly perfected. This damages our ability as societies to function in a healthy way. Our capacity to navigate hard problems is diminished by the ever decreasing influence of hard science on politics and policy.
> Without becoming an expert yourself it's very difficult to know who or what to trust.
Who says there is anyone who can be trusted? People keep looking for leaders they can trust and it takes only a brief look at history to see that the search won't stop despite the jaw dropping futility of the exercise.
The important thing is to check that people have incentives to tell the truth and no conflicts of interest. I'd trust someone untrustworthy if they were making money off my well-being. The only thing to watch out for is them not being forthright about their incentives.
We shouldn't trust that skyscrapers stay up because engineers are trustworthy. They stay up because the engineer goes down with the building.
Completely agreed. While I strongly believe in citizen science and people's right (and perhaps obligation) to critique established science, there is just so much poor analyses done by people to criticize some scientific findings they do not approve, motivated emotionally or otherwise. This phenomenon does not bring us closer to finding scientific facts or resolving the replication crisis. People should learn some basic analytic tools first.
I can see why this has been downvoted, but I didn’t mean to sound like a luddite. I really just think there is a lot of low quality research, especially in the social sciences, done primarily to keep an publishing schedule up for pressured academics. We’d often be better to spend more time thinking about and planning fewer better studies.
One way that journals could actually add value (a concept to which they seem resistant) would be to review the statistical analysis. Statistics is hard and easy to get subtley wrong, and is often an independent skill to the underlying science. If journals had statistics experts to critique the analysis techniques prior to publishing it would be a great improvement in the confidence in which we could read papers.
I think this is a great idea, but in my experience there are surprisingly few such experts available. In practice, most statisticians not doing active methods research (I'm thinking of 'trial statisticians' mostly here, in CTU's) just cargo cult whatever procedures previous trials used. I guess they would pick up issues around sample size, but without also integrating that with some substantive knowledge about plausible effect sizes I'm not sure what value they would have.
Plus, it would reduce the number of publishable papers quite substantially including from high profile authors/groups, so I don't think they want the fight. We should also remember that most journal editors are also involved in publishing this research — they often have no real incentives to make things awkward.
>The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs.
That's a very simplistic take on it. Bad science is a necessary part of the process and dealt with accordingly by the scientific method.
The problem is that science that is bad or incomplete is being reported as fact or truth, or arguably even worse, as entertainment in order to gain an audience. This is what actually eroded the trust in science, as people kept repeating things that reinforced and further misshaped their biases.
Human nature tends to distrust stuff we don't understand. Hence, our trust in science, which in many fields is often beyond the understanding of laymen, have to have constant reinforcement. However, the goal of media, and especially social media is to increase eyeballs for their content, and the truth sells a lot less well than sensationalised content pandering to the audience.
Simply put, there is not much profit in reporting science truthfully, and every incentive to sensationalise it.
part of the issue is that producing shit research cost very little, and debunking shit research cost a lot. just pick up one of the many "x reasons why earth is not a sphere" videos, some are pretty easy to debunk, but other require to understand i.e. potential fields (if earth is round why don't train engineer take into account curvature when laying tracks and variations thereof)
Shit research can sometimes be debunked just by running the numbers given by the shit researchers. Flat earth theorists don’t often offer up actual research (and the one group I’ve seen try got negative results and concluded they messed up, not that they were wrong).
A theory ought to be able to answer questions like “Why don’t train engineers take the curvature of the Earth into account?”
The problem is when someone comes along and thinks an unanswered question (or even just someone not knowing the answer off the top of their head) proves a theory is completely false (or worse, proves their favorite theory correct). (And to even believe the Earth is flat is to be a conspiracy theorist so in this particular case no prove will ever suffice anyway.)
I agree, but I don't think this is limited to science. I think a great deal of everything is in fact trash. This is why we need education and good faith discussion.
Science itself is the best method we have for exploring and making sense of the world around us. The method is rock solid.
In between us (gen pop) and The Method are scientists, and scientists are just as fallible as any other group of people - lawyers, politicians, coders, shop assistants.
In other words, emphasize the process more than the outcomes. If the scientific process used proves sound, then I have more confidence in the outcomes.
Alas, that's a hard sell to laypersons thru the mediums of soundbites and tweets.
What about the replication crisis? It's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods). I can't tell you how many times I've seen headlines or abstracts of studies that seem to contradict ones I've seen previously, and back and forth! Particularly in the social sciences.
I recall one study that said all white people are committing environmental racism against all non-white people. I dove in and read the whole thing wondering what method could have yielded scientific confidence in such a broad result. Turns out the model used was a semi-black box that required a request for access and a supercomputer to run. But it was in a Peer Reviewed Scientific Journal and had lots of Graduate Level Statistics so I guess it seemed trustworthy.
The issue here has not much to do with the replication crisis. It has to do with the fact that most people who use bits of information to make their point more convincing don't care whether that information is true or not. They are not seeking to convince the other side of the issue, they are seeking to convince other believers.
It is literally like this:
- someone makes a point that questions your believe
- you google a phrase that would come in studies that proof otherwise
- you take the first thing that looks promising, and fly over the first page, and paraphrase a good bit in a way that makes your point
- you publish it as part of a post, youtube video or whatever
- danger averted
Bad studies play into this, but even if the studies are good, or bad studies that have been retracted the same thing happens. James Wakefield who originally published the "combined vaccines cause autism study" after patenting a non-combined measles vaccine had his study retracted by the lancet soon after publication. He lost his status as a doctor etc. And you will still find people who use his study as a source.
Of course studies whose outcome collide with our believe systems are always harder to trust than those who validate it — but this is why you look at the methods used and other indicators that might make that study bogus.
A replication crisis indeed exists. All the more reason to analyze rigorously. Poor analyses (and borderline name-calling) in the original article do not help with the crisis.
>it's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods).
I don't think this is true. It is possible to put a lot of work into unsound statistics and to make a lot of "noise and fury" about how mathematical you are while failing some basic principle, but I don't think sound statistics can mislead. The replication crisis was caused by scientists not being rigorous and journals not forcing them to be. You absolutely cannot accept publication as a sign of sound techniques except in journal/field combinations that have a deserved reputation.
Of course they can, unless you magically exclude all statistics that made a bad assumption on independence.
I plot all the daily high temperatures and the presence of the ice cream cart and it turns out the ice cream cart causes warmer highs! Solid statistics.
Turns out the guy that has the ice cream cart has a weather app on his phone though and doesn’t come out on forecasted cold days.
Is that the fault of statistics though, or the non-statistical implication of causation that was tacked on the end of the statistical detection of correlation? Statistics is pretty explicit that it can't tell you about causality, right?
> It's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods)
The book "How to lie with statistics" is one of the best statistics textbooks that I have read. It basically makes you immune to misleading stats (charts, tables, everything).
IIRC, the only thing that is missing from the book (it's a really old book) which is very relevant is p-hacking.
Given the explosion in the number of journals and the impossibility of effective peer review, being published in a journal does not mean what it used to. This is part of the material drivers for the replication crisis (journals can no longer effectively gatekeep scientific validity), but it also reflects something real about the practice of science: little social cliques come up with pet theories and, over time, "fight" with these theories on epistemic common ground. The successful ones, we'd like to think, are the ones that last the most rounds in the fight, but that probably only holds in the long run. Contradiction, in itself, is normal (and was before!)
> That said, I give the benefit of doubt to the author of "The DK Effect is Autocorrelation." It is a human error to be overly zealous in some opinions without thinking it through.
If only there were a term for "a cognitive bias whereby people with limited knowledge or competence in a given intellectual or social domain greatly overestimate their own knowledge or competence in that domain relative to objective criteria or to the performance of their peers or of people in general"
That happens when science is politicized, and any scientists critical of the “official” results is destroyed. From climate to Covid, so many areas where that happens.
No, I mean researchers and professors suddenly not getting any research grants anymore, suddenly getting fired from their tenured jobs, not being invited anymore to conferences, etc.
You could build your whole career from the 50's on discrediting the link between smoking and cancer.
The Koch brothers were (and the current brother and estate of the other) are happy to write economists checks to prove laissez-faire capitalism under a libertarian government is the best system. Or that climate change isn't real.
If you're willing to generate evidence climate change isn't real, Exxon etc have some nice checks for you.
If you're willing to show how corn syrup is good for American than the Iowa farmer association has money for you.
It still seems to me like "The DK Effect is Autocorrelation" is basically correct. The important thing isn't whether or not independence should be the null hypothesis, because calling something a "null hypothesis" is just an arbitrary label that doesn't affect reality. The important thing is that what we can actually conclude from the Dunning-Kruger paper is a lot less than popular presentations of the concept claim. In particular, "more skilled people are better at predicting their own performance" is really not supported by the paper, since that's not true of random data, which has everyone being equally terrible at predicting their own performance. If the random data can reproduce that graph, then the graph can't be proof that more skilled people are also better predictors.
Anyway, "The DK Effect is Autocorrelation" definitely seems to be both statistically literate, and a good faith criticism of the Dunning-Kruger paper. In light of that, calling it "anti-scientific" seems unfair, since criticism and debate are an important part of science.
> calling something a "null hypothesis" is just an arbitrary label that doesn't affect reality
It does affect your conclusions though.
The choice of null hypothesis in "The DK Effect is Autocorrelation" determined how the random data was generated. The hypothesis is: "nobody has any clue whatsoever how competent they are". The random data was specifically crafted for that hypothesis.
The choice of null hypothesis in this article is: "everyone roughly knows how competent they are". This random data, too, is specifically crafted for the null hypothesis.
So what does this mean? If you pick the a particular null hypothesis then you can try to argue that the DK is a statistical artefact. But it's not, it is an artefact of choosing a particular null hypothesis.
No, nulls matter a great deal. If you want to test a claim in Null Hypothesis Statistical Testing, the "significance" of the claim is in direct reference to the null. Changing a null will change the significance of the alternative. My favorite statement of this is from Gelman:
> the p-value is a strongly nonlinear transformation of data that is interpretable only under the null hypothesis, yet the usual purpose of the p-value in practice is to reject the null. My criticism here is not merely semantic or a clever tongue-twister or a “howler” (as Deborah Mayo would say); it’s real. In settings where the null hypothesis is not a live option, the p-value does not map to anything relevant.
I think what contributes to this phenomenon are both second-option bias[1] and motivated reasoning, at least with respect to those who choose to believe in the poor analyses.
I read a lot of papers on behavioural economics and psychological decision making experiments for university, like dunning-kruger, kahneman, etc and in my opinion the first autocorrelation article reads like a rebuttal paper but more informal, the approach is scientific even if it may be flawed. This is how knowledge advances. I disagree that it is anti-science. Challenging accepted postulations is good. Even famous professors make mistakes, I don't blame the writer for making an honest mistake. That's how we got this new piece of writing
Behavioural science is a pretty new field, its pretty easy to get abberant results or manipulate the results to show 'something' statistically. Many findings in earlier papers could not be replicated, or had applied statistics incorrectly, or showed different results when research participants were not white college kids.
This is a whole other problem within academia, the pressure to publish something even when there is nothing and perceived legitimacy based on the number of citations a paper has. My professor always said don't look at the number of citations, understand the method and the rebuttal, there were numerous low citation but solid papers showing flaws in famous ones but everyone who isn't deep into the subject holds the original assertion to be legitimate because its "famous"
Most social science is shoddy, fake, or otherwise misleading (i.e. it proves nothing meaningful despite the claims of the researchers). If you believed every social science study you heard about, you'd be more wrong about the world than if you disbelieved them all.
> The anti-intellectual, anti-scientific streak in many poor analyses claiming to debunk some scientific research is deeply concerning in our society.
People endlessly reference the Dunning-Kruger effect as a meme, without ever having read the paper, let alone having checked its methods. You don't seem to have a problem with that.
On the other hand, after seeing an article that uses essentially statistical arguments to debate a scientific study you conclude that there is some "anti-intellectual, anti-scientific streak" in our society and that it should be of grave concern.
This doesn't make any sense except as an extreme case of virtue-signaling.
Seems quite reasonable to argue that superficially plausible "debunkings" by people that apparently misunderstood a paper are more harmful to scientific progress than people casually referencing the scientist's names as a meme or insult. (And I say that as someone who didn't think the DK "debunking" argument was totally without merit)
What's more harmful to medicine: a fashionably non-expert contrarian who doesn't understand the appropriate null hypothesis making a superficially plausible statistical argument that actually the trials suggest the drug is harmful to wide acclaim from laymen, or people casually referencing or even being administered the drug without reading the original trial writeups for themselves?
> Why so angry? I know I’ve taken this far too personally. I have no illusions that everything I read online should be correct, or about people’s susceptibility to a strong rhetoric cleverly bashing conventional science, even in great communities such as HN. But frankly, for the last few years, the world seems to be accelerating the rate at which it’s going crazy, and it feels to me a lot of that is related to people’s distrust in science (and statistics in particular). Something about the way the author conveniently swapped “purely random” with “null hypothesis” (when it’s inappropriate!) and happily went on to call the authors “unskilled and unaware of it”, and about the ease with which people jumped on to the “lies, damned lies, statistics” wagon but were very stubborn about getting off, got to me. Deeply. I couldn’t let this go.
I am afraid I actually agree with the author's point. The anti-intellectual, anti-scientific streak in many poor analyses claiming to debunk some scientific research is deeply concerning in our society. If someone is trying to debunk some scientific research, at least he should learn some basic analytic tools. This observation is independent of whether the original DK paper could have been better.
That said, I give the benefit of doubt to the author of "The DK Effect is Autocorrelation." It is a human error to be overly zealous in some opinions without thinking it through.