Amazing! Saves so much time not having to read a paper only to find there is really no code or very limited code to run. Not having delved too deep into this a suggestion would be to add the level of completeness of code to each paper. There are times a paper has great code included and 80% is there but is missing a crucial piece, sometimes the secret sauce, which would render it if not impossible to use. This is the case with a lot of OpenAI papers.
Excluding some (e.g. OpenAI) papers that have limited their released results for ethical purposes, why would the authors generally not include their code?
I remember going through a digital image processing course in uni where the final project was to implement a paper and check the results, and I remember that our results, when coded were different from the paper’s authors (although I can’t remember if it was because we didn’t code it like they did or if their results were not to be trusted).
It’s just so frustrating and borderline disingenuous to publish results, mention bits of code, but not include the whole code.
> why would the authors generally not include their code?
Decades of "printing out the code listings would cost too much paper" made many researchers get comfortable with the idea that nobody would ever see the code behind their work. That means:
- your code can be shit and potentially buggy and nobody will point and laugh at you
- replication is harder, so you can bluff some of the details
- if key findings are actually bugs, nobody will find out
These are all terrible reasons, but they're comfortable reasons as well. Open access was the first barrier, code included is simply the next.
If code included would've been commonplace, fiascos such as Imperial College of London's possibly-totally-bogus covid simulation [0][1] would've never made it to the cabinet.
