Hacker News new | past | comments | ask | show | jobs | submit login

I think this can be very field dependent. I am a physicist and perhaps I am overly optimistic. However, I find that the existence of supplemental materials is helping things a lot. Generally, in an paper (in experimental physics), you are trying to tell a story about your measurements and the understanding that you gain from them. So, it's nice to be able to tell that story, where you focus on the most relevant data. However, with supplemental materials, you can add a lot of the sanity checks that you did (sometimes at the prompting of referees) in the supplemental materials. For example, maybe I want to tell a story about how in some material that by applying a voltage, I can influence the magnetic properties of that material. There are a lot of baseline measurements that you perform these days to determine that. In the past, with page limits, you couldn't include all of that information, but now, you can place a great deal more in.

In my particular field, my raw data is online, so that can be checked--though without more meta data, it would be of limited use to someone without my notebook and annotations. A lot of the code for the reduction tools that we use is also open sourced as well. There have been some moves in science towards reproducible data analysis, but the problem is infrastructure/ecosystem. For example, let's suppose that I use a python code to do some statistical analysis on my data, it could be hard for someone else to try to reproduce that say 20 years from now--because they won't just need the library I wrote, but the entire ecosystem that it lives in--I don't have a good answer for that.

But, I think that for high profile results (again, I'm optimistic) in physics, there's an incentive for other groups to try to reproduce them to see if they hold or not. There have been cases where someone honestly thought that an effect was there (and it was reproducible), but it was found out later that it was due to something extrinsic--for example, many sources of Boron that you buy commercially have trace amounts of magnetic impurities, so it took some time before someone realized that this was the cause for a weak signal that people were seeing.

In some communities, such as crystallography, you have to submit a file which shows your results and it is automatically checked for chemical consistency. I think this can help weed out some sloppy errors. But, it is still possible to make mistakes.

Also, with journals like Nature Scientific Reports, it becomes feasible to publish results that aren't so exciting, but are technically correct (it takes a lot of time to write a paper and an even longer time to publish it and the cost benefit analysis makes it difficult at times to publish everything, so lowering the barrier to publication to technical correctness rather than excitement helps people to publish null results so other people don't waste their time).

There's also the question of where to draw the line as a referee. If someone is publishing a methods paper where they have made a new analysis technique that is going to be implemented in software that most people are not going to check, then I feel obligated to check their derivations, reproduce integrals, etc. For other papers, looking for consistency between different measurements and the literature is probably sufficient.

There's still a lot of work to do, but I don't think things are completely disastrous in physics. I recently had a colleague retire and he was very proud of the fact that his data had never been wrong. The interpretation might have changed with time, but the data itself was right. I think that's the best we can hope for...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: