I often have my PhD students re-implement ideas to gain skills and to verify a method works. It is possible to publish these efforts, but it isn't easy. It typically involves comparing multiple methods on datasets they haven't been tested on before to see how well the results generalize beyond the original paper. It is hard to publish in prestigious venues with this approach, but we have had some success. Replication and comarison makes for a good MS or early PhD project, but later PhD students have to showcase an ability to create new algorithms. In my lab, we try very hard to not cherry pick and to do good work that generalizes.
From your profile, you are working in Deep Learning. In other fields it's much more difficult to be sure that the replication is accurate.
Do you have the correct variety of rats? Are they receiving the same kind of food? Does they get the same illumination during the day? ... Theoretically al the details should be clear from the published paper, but most of the times the paper is full of "underspecified"[0] parts.
Also, in many fields even if the paper is only about calculations in a computer, the programs are not published (or are a mess (or an unpublished mess)) and the data are not published (or are a mess (or an unpublished mess)). So
it's more difficult to even make a direct copy of the results of the paper.
Also, there is a lot of informal replication, essentially what you do but without the final publication step. Just replicate somewhat similar to the original paper, but then publish a version with some extension or tweak.
> Replication and comparison makes for a good MS or early PhD project
Do you think that might be the solution? That is, to get an MS your final project has to be an attempt at replication, and then a PhD has to be a new contribution. If that became the standard, would it solve a large part of this whole issue?
There is a huge difference between physical and computer sciences. I agree that it's an excellent use of time for new students if the main factor for the work is time/salary. This paper was aimed towards medical/bio/clinicians and when you add material costs, the variability of biology and multiple people required to run large experiments everything falls apart. One of my PhD projects (Primate neuroscience and new medical devices) took 4 years2 grad students2 staff6 animals (only 2 made the paper)animal housing costs for 4 years + ~$150k in materials. And there are only ~10 research labs in the world that have the ability to do this type of research so no one is going to . We see a lot more replication via extension - "this theory worked for this group, what if we take that as true and extend it in a new direction, where does the science fall apart there?"
I'm not arguing that replication isn't important and a lot of false positives get through into the literature but without 10x-ing the research budget and infrastructure as well as changing the publication incentives there isn't going to be any real movement.
