> When I do code reviews I read the tests first. How can you review code with no tests?
Please grow out of this religion if you care about robust, secure and well written software.
You can look and see what are the types of things that get fixed in OpenBSD. Tests are not going to tell you that code works around safety measures built in to the OS. Tests are not going to tell you that the code uses time as entropy to seed RNGs. Tests are unlikely to tell you about the subtle integer overflows or out-of-bounds accesses that attackers can exploit. Tests are unlikely to tell your daemon will ungracefully fail due to fd exhaustion while serving a larger amount of requests on a loaded system. Tests are not going to tell you there are deceivingly familiar looking functions with surprising return values that will confuse people. Tests are not going to tell about bad style and code smell that makes life harder for anyone working on or looking at the code.
There are so many things that are hard or impossible to test, but which are vital to handle right if you're doing anything robust and secure. In fact you could construct a test for some of these things, in theory, but in practice you never will. And you certainly never will unless you have actually read the code, because all these things are issues within implementation details inside the black box; you must read the code (and understand the underlying platform) to even become aware of these issues.
Sure, tests are really useful for catching some specific issues. But there is so much more. Tests are somewhat like compiler warnings: often extremely helpful, but even totally insecure crap can compile cleanly. And it is possible to write good code even if you're not using the smartest compiler to compile it.
If you wish to learn how to review code with no tests, I recommend you read the C language chapter from The Art of Software Security Assesment[1]. Actually read the entire book. Then go through the commit logs in some project -- such as OpenBSD, who've been doing it for almost 20 years -- and see what kind of issues they find and fix.
On the other hand, there's more to testing than traditional unit testing in which a programmer enumerates a series of cases and verifies that what they think should happen (which may or may not even be correct) happens.
There's fuzz testing. There's QuickTest-like testing, which is tricky in this sort of situation, but powerful if you take the time. There's static analysis, which are basically a form of automated testing. In fact, static analysis can in fact test some of the things you claim can't be tested. Some of the other things are also more testable than you are saying, such as resource exhaustion, which can be both simulated via injection or via simply exhausting resources in the test case.
Further, even the relatively-weak guarantees that traditional unit testing can provide are a foundation for further analysis. What's the point of analyzing a bit of code for whether or not it uses entropy correctly if you miss the fact that the code is straight-up buggy? Given the demonstrated difficulty of writing code even before it's security code without solid testing, this is hardly the time to be claiming that testing isn't necessary.
This strikes me as another iteration of the Real Man argument ("Real Men write in assembler" being one of the originals). The truth is we need all the help we can get because we humans aren't very good at programming in the end... and of all the places I'm not inclined to accept the Real Man argument, it's in security code. This is the sort of code that ought to be getting covered by two or three separate static analysis tools, and getting reviewed by skilled developers, and getting fuzz tested, and getting basic unit tests.
No Real Men arguments here. I did not claim testing isn't necessary. I just wanted to challenge the blatantly religious attitude of pretending or believing that a particular form of testing (one with clearly written tests the poster above me would actually read) is all there is and that you cannot review code otherwise. You definitely can, and must.
As far as I am concerned, all techniques that help improve code are welcome and even necessary. This includes testing, manual review, static analysis, etc.
Please grow out of this religion if you care about robust, secure and well written software.
You can look and see what are the types of things that get fixed in OpenBSD. Tests are not going to tell you that code works around safety measures built in to the OS. Tests are not going to tell you that the code uses time as entropy to seed RNGs. Tests are unlikely to tell you about the subtle integer overflows or out-of-bounds accesses that attackers can exploit. Tests are unlikely to tell your daemon will ungracefully fail due to fd exhaustion while serving a larger amount of requests on a loaded system. Tests are not going to tell you there are deceivingly familiar looking functions with surprising return values that will confuse people. Tests are not going to tell about bad style and code smell that makes life harder for anyone working on or looking at the code.
There are so many things that are hard or impossible to test, but which are vital to handle right if you're doing anything robust and secure. In fact you could construct a test for some of these things, in theory, but in practice you never will. And you certainly never will unless you have actually read the code, because all these things are issues within implementation details inside the black box; you must read the code (and understand the underlying platform) to even become aware of these issues.
Sure, tests are really useful for catching some specific issues. But there is so much more. Tests are somewhat like compiler warnings: often extremely helpful, but even totally insecure crap can compile cleanly. And it is possible to write good code even if you're not using the smartest compiler to compile it.
If you wish to learn how to review code with no tests, I recommend you read the C language chapter from The Art of Software Security Assesment[1]. Actually read the entire book. Then go through the commit logs in some project -- such as OpenBSD, who've been doing it for almost 20 years -- and see what kind of issues they find and fix.
[1] http://ptgmedia.pearsoncmg.com/images/0321444426/samplechapt...