This is a cool idea but seems fairly limited when you take into consideration the state of the resources used in your application. For example, if I have a method that checks the balance of an account, I may hit that method with an account with a positive balance, make a payment, then hit the method again and have a zero balance. The method might take an account instance and the same account instance yields two different results.
I realize that this may be an issue with a poorly written method or un-anticipated side effects, but with legacy code, that is often what you have to deal with.
Yeah, this will be great for a functional language. PHP functions tends to have side effects and their results often rely on variables outside of the function which means the tests will be wrong.
not just globals, but what could be in the database, or global variables get, post, session variables. you don't test with your production database do you?
This is why unit tests suck. They are but a simple exercise in basic code interpretation. A computer can easily do it.
What you want to test is the interface of your module to the rest of the application and whether it satisfies the published contract, and preferably if it handles weird cases correctly.
> What you want to test is the interface of your module to the rest of the application and whether it satisfies the published contract, and preferably if it handles weird cases correctly.
I don't know what's a "module". If a module is a "small unit of code" (eg, a class), that's usually what you cover in your unit test.
Bleh. That definition of unit tests is so bland as to not differentiate between any kind of testing. That definition explicitly states that having a person (like QA?) go through the app and see what happens also constitutes unit testing. And that's bullshit, manual testing is not unit testing. And neither is testing an entire module at a time.
Most definitions define unit testing as testing the "smallest piece of code that can be tested in isolation". Then my criticism definitely applies.
And given my code reviews, it's obvious that the vast majority of programmers see unit testing as testing every individual function/method in isolation. This actually hides problems by increasing total source code (actual code and the tests + "compromises" for testability) and locks down internal structure of supposedly encapsulated logic. Both are very, very bad things.
It is designed to generate input for your code (the public surface by default) to determine whether it runs correctly. It can use code contract features in .Net to help inform whether your code is correct. It is great for finding those 'weird cases', and where you find a particularly interesting case, it is easy to add it as a test in your suite.
You can see it used (and play around yourself) through the browser here: http://www.pexforfun.com
You're probably testing too much. Take Kent Becks advice and only write a test when you can bet a dollar that it would fail. Don't write tests for things which you never fail at. BTW, the term "unit test" can mean many things depending on who you ask. I prefer to stick with the general term "test".
This works until you make a significant refactoring. If you have great coverage, you are pretty sure everything's jiving correctly. Not sure what kind of refactoring tools existed back in the SmallTalk days for Beck.
The OP said that unit testing sucks because it can be automated. If your tests are so simple that they can be automated I would argue that they are useless, because you're not really testing functionality, but more boilerplate. You can usually skip underlying boilerplate tests if you test the functionality (within reason). Focusing on the right level and skipping the lowest level actually makes refactoring easier, since you have a degree of freedom at the very bottom left.
I realize that this may be an issue with a poorly written method or un-anticipated side effects, but with legacy code, that is often what you have to deal with.
Cool stuff, though!