> However, the reliability of the average publication has probably decreased. If there are indeed "diminishing returns to science" in recent decades, as Patrick Collison and Michael Nielsen argue [2], with roughly constant rates of important discoveries (as rated by experts) and flat economic productivity (a measure that we'd expect to correlate with technological progress) despite exponentially growing numbers of scientists, publications, and dollars devoted to science, then the quality of the average paper, scientist, or dollar allocated to research must have gone down. In that case, a randomly chosen older paper should be more trustworthy than a newer paper.
I don't quite follow this line of reasoning. I must be misunderstanding something because Sarah is way smarter than I am. But it seems to me the diminishing returns in biological science are due discoveries having a smaller impact, not on the quality of the paper. i.e. a 1985 paper doesn't have more of an impact than a 2019 paper, but a paper released in 1985 has a bigger impact than that same paper released in 2019.
If older studies are more fruitful than modern studies I would imagine it's because they were wackier. Modern studies have to deal with a more restrictive IRB and also seem more likely to fit inside the box. I hardly ever heard of modern studies where someone was like "I hooked up a battery to a weird part of the brain to see what happens."
I have a theory that as colleges, PhD programs, and then professorships have gotten more competitive that we have been selecting for higher and higher conscientiousness which has led to greater conformity. My professors didn't seem nearly as wacky and they described their professors/mentors.
> I don't quite follow this line of reasoning. I must be misunderstanding something because Sarah is way smarter than I am. But it seems to me the diminishing returns in biological science are due discoveries having a smaller impact, not on the quality of the paper. i.e. a 1985 paper doesn't have more of an impact than a 2019 paper, but a paper released in 1985 has a bigger impact than that same paper released in 2019.
Less competition leads to less p-hacking, less pressure to publish or perish leads to greater risk taking in experiments, fewer proximate measures of impact lead to more confidence that results are real rather than an artifact of finding some intermediate measure that looks significant, and just random fashion leading to some areas going out of style that would actually have been fruitful if pursued further.
I guess I always assumed that ignorance was a more important cause of p hacking than cheating. But if it's mostly cheating I could see a high pressure environment leading to more p hacking.
If that's true that would mean that a great deal of professors are dishonest.
Rather than 'dishonesty' per-se, it is probably closer to 'wishful thinking' most of the time, with a side-order of 'don't look a gift horse in the mouth'.
I don't quite follow this line of reasoning. I must be misunderstanding something because Sarah is way smarter than I am. But it seems to me the diminishing returns in biological science are due discoveries having a smaller impact, not on the quality of the paper. i.e. a 1985 paper doesn't have more of an impact than a 2019 paper, but a paper released in 1985 has a bigger impact than that same paper released in 2019.
If older studies are more fruitful than modern studies I would imagine it's because they were wackier. Modern studies have to deal with a more restrictive IRB and also seem more likely to fit inside the box. I hardly ever heard of modern studies where someone was like "I hooked up a battery to a weird part of the brain to see what happens."
I have a theory that as colleges, PhD programs, and then professorships have gotten more competitive that we have been selecting for higher and higher conscientiousness which has led to greater conformity. My professors didn't seem nearly as wacky and they described their professors/mentors.