"If you're a Java programmer and want to have a rude awakening, go download Jester. Jester is an automated mutation testing tool - it goes in and replaces "<=" with "<", "&&" with "||", "!" with whitespace. And then it re-runs the tests. And then you get to watch in horror as your tests still all pass, regardless of what the product code actually does."
Heckle is a mutation testing framework for Ruby. I don't use it all the time, but my tests are definitely better now that I've tried it and seen what some of the edge cases are...
Normally they really shouldn't, but if you have a problem writing test cases (which probably means you have a problem programming to begin with) then you may not be testing with enough variation in your input to detect corner cases, tests that should come up false etc.
An example:
int fun(int a, int b) {
if (a >= b) {
return 1;
}
return 0;
}
now you could test this by saying:
fun(4,3) should give '1' as the output
and
fun(3,4) should give '0' as the output
But you'd miss to test for equality, with
fun(3,3) which should *also* give 1 as output
So you'd need an extra test for the '=' part of the condition. Replacing '>=' with '>' will still pass the first two tests, so if you forgot about the third you will not even realize something is wrong until your code starts breaking in mysterious ways.
Unit tests have to be just as good as the code you wrote.
More like "bad unit tests are useless, good and useful unit tests are hard and take a significant chunk of time, therefore weigh the costs vs the benefits before deciding whether to bother or not"
Hm. Before deciding that good and useful unit tests are too expensive, spend a week or three doing live support of poorly reported and hard to reproduce issues where the numbers were slightly out or the result came back ten seconds later than it should, one time. that will change your perspective on what's hard and what's not. And possibly damage your will to live.
If the bugs are that subtle what makes you think you're unit tests are comprehensive enough to catch them? The hardest bugs to debug are the types that show only on high latency networks and only when dealing with files bigger than 2GB and only after the app has been running for at least 50 hours. Those are also the type of bugs which are almost impossible to write unit tests for.
I'm not arguing against unit tests, they definitely have a time and a place. I'm just not convinced that they are an easy and instant cure for all your hardest bugs.
Unit tests may leave you with remaining subtle bugs. This is better than having both subtle and obvious bugs. Obvious under test, e.g. "the class returns null when it should return object y", which can have subtle and easy-to-miss symptoms in a production system
It shows that green bars on all your tests don't necessarily mean anything. All too often, developers don't test for the cases that may actually fail (because, obviously, their actual code is only correct for the cases they're cognizant of).
Because they are arbitrary and not a one-to-one correspondence with what they purport to be testing. One false positive and everything could still be broken.
Never tried it. But its probably because unit tests can only tests things you explicitly test for and if you don't have your corner cases covered, then these changes wouldn't blow up your tests. Someone correct me if I'm wrong.
Not a fan of unit testing my self. I feel the approach lacking.
I don't see why a fanatical anti-rant is any better than a fanatical rant.
There's this theory out there that Agile projects can refactor fearlessly because there's this immaculate suite of tests that can sound the alarm the second the smallest regression gets introduced. But anyone who's actually tried it knows that it's mostly just a fantasy.
Poppycock. My project has an extensive test suite that we rely on to make major design changes with alacrity, just exactly the way that unit-test advocates advocate. We couldn't dream of doing what we're doing without it.
It's true that not every module of every system responds equally well to this style of testing. It's true that putzing around revising test infrastructure for the nth time really sucks. But these are points that belong in a serious discussion, the kind that recognizes tradeoffs.
cashto's post was a "serious" effort to disperse the cloud of dogma surrounding unit testing and TDD. I hardly think that lashing back with "poppycock" is driving the conversation in the direction of "serious discussion".
cashto emphasized that unit testing is useful for teams with non-expert programmers (viz. his comment about the Dreyfus model of skill acquisition). Many of us are non-experts who benefit from working with the TDD training wheels on; it's O.K. to be less a Jedi All-Star hacker. But the phenomenal programmers, the men and women who were 10 times better than me, could not be bothered with TDD, even through the endorsed it for mere mortals.
To paraphrase the TDD credo of “no test is worse than having no tests”, we can assert that blind faith is not better than no faith at all.
The training wheels analogy was the weakest part of the article to me. TDD isn't a teaching tool, and the fact that they useless to people in a very specialized niche doesn't really say anything about their general value.
I also notice reference to writing tests after the fact. I understand TDD to involve writing tests first. That's how I practice it anyway.
That's just poisoning the well. Yes, X is fine if you're an idiot is exactly what you would say if you didn't want to have a conversation about something and would rather shut down discussion.
this is one of his worst arguments. I find it almost exactly the opposite. I've been programming for almost 20 years and I am considering myself an expert. and many people around seem to agree ;).
most of the beginning was spent w/o any kind of unit testing. But I find it that I do more and more testing with time, and I'm sure as hell I'm not becoming a weaker programmer that suddenly needs the training wheels ;)
> My project has an extensive test suite that we rely on to make major design changes with alacrity, just exactly the way that unit-test advocates advocate. We couldn't dream of doing what we're doing without it.
My experiences are the same as yours. I've refactored code where the unit tests caught that the new structure introduced subtle logical inconsistencies in the program. The unit tests meant I caught this immediately; without them I would have had to wait until subtle and hard-to-fix bugs emerged.
On a different project, I've been able to play around with the structure of the code, secure in the knowledge that my regression test suite will tell me when I fuck things up.
Tests, when done well, are a lifesaver, and essential to any large complex codebase.
> But these are points that belong in a serious discussion, the kind that recognizes tradeoffs.
There's this theory out there that there are creatures that aren't birds but somehow fly through the air using wings. Anyone who has looked for these mythical "bats" knows that they are just fantasy.
Here we see the risk of asserting something isn't real just because we haven't ourselves seen it. I've held a bat in my hands, and I've had my butt saved countless times by tests I've written.
Ah, when I first began to read this, I expected one of those rants against unit tests which don't leave any kinda good thing about tests and I mentally got ready to comment about that.
However, I am quire pleased that this is a very sensible and mature position about unit testing (I think a short version would be: well, testing can help, it won't carry you and you have to stand on your own feet, but it can help), which is pretty close to my own position. Good Article.
Aside on testing: I was surprised in re-reading Fred Brooks how cloose he sounds to unit-testing, in Mythical Man Month, 2ed, pp.147-148. He suggests testing components separately, before integrating, even though this requires lots of "scaffolding".
Scaffolding includes dummy components, miniature file (minimal eg of a file format), "generators of test data, special analysis printouts, cross-reference table analyzers" (jigs and fixtures).
"It is not unreasonable for there to be half as much code in scaffolding as there is in product". I end up writing a lot of the above kind of code - not as automated unit tests, but just as a way to make sure my code is doing what I think it's doing - just plain ol' testing. It can feel wasteful, so it was reassuring to hear Fred's take.
As with all things, truth is somewhere in the middle. I've found that unit tests do help me write better code and find corner cases which I would otherwise miss. But I don't treat them as a religion, and I don't write them for every function.
You'd be amazed at the number of experienced developers who tell me "it is best practice to do XYZ" who when challenged as to what is the underlying justification for this recommendation can't actually provide a sensible answer.
I've tried TDD for one project and it gave me a nice feeling of having a safety net. It was a good feeling. The problems started to appear when the project increased in complexity and needed a redesign in some of it's parts. I had to do _again_ twice the work to have the tests match the code behavior.
The article mentions this problem too. The benefit of the tests disappear when your code changes beyond refactoring.
With my students and used the analogy of unit tests (or asserts) with a fisher net to catch bugs. As a fisher net, it is made of a mesh that won't be able to catch small fishes. It may also have big holes in it here and there. So there is always a limit to its efficiency.
It requires a significant effort to make unit tests more efficient. This effort must then be balanced with the potential costs of the bugs and across the application, because some part of the program are more critical than others.
A good programmer should thus balance his effort in unit tests writing.
I also suggest to write unit tests as needed to be more efficient. If the software is a data processing pipeline kind, simply write a very tight data validity checks at the end of the processing pipe and inject test data (including bogus one). If no problem ever show up with the test data, there is no point in writing unit tests for all intermediate steps of the data processing pipeline. If a problem is later discovered, then write the test, but only at the middle of the processing pipeline. You'll know if the problem is before or after. Then continue as needed. With this approach, you'll write unit tests only where it is needed and won't waste you time.
Suggesting as the author that we don't need to write unit tests is like saying we don't need to put the car safety belt because people still die in car accident with it, and most of the time we don't have any accident even if we don't wear it. Is this a smart advise ? I don't think so.
BTW, I use asserts wherever I can by reflex in my code because it saved me valuable time on many occasions.
How about "buckshot testing"? I wrote a file system test that made every single API call with null arguments, good arguments, illegal arguments and then logged the error code. We ran this against every release of the OS and simply dif'd the result against the previous release run. It not only aided in bugfinding, it identified necessary release notes about API differences. The bigger point is, you have a computer at your disposal. It can make LOTS of calls in a short time. Try calling that function with every conceivable argument. Once you have a baseline, a diff tool can reveal everything.
"Many of them will not even survive the process at all."
That's the point, and I think everything else in this article stems from it. Unit tests are supposed to be transient; you're supposed to be able to just ditch them and replace them when you ditch and replace the code they're testing.
Functional tests, on the other hand, should be longer-lived.
Personally, if anyone asks me if they should unit test, I'll play the fanatic and take a hard line. Mostly because the way to learn the usefulness of unit tests is to write them. You quickly learn what they're good at and what they are bad at.
Once you've written enough of them, you kind of seem to develop an instinct as to what unit tests are useful for.
I'm not sure how valuable these articles - or threads - really are for anyone. Don't we have to balance cost vs. value for _anything_ we do? Find and replace "unit testing" with "documentation" in this article and we could be having the same conversation.
The article doesn't seem to be claiming that unit testing is bad, rather that religious promotion of it is bad, and the benefits and drawbacks should be weighted up like any other decision.
Obviously SQLite has got a tremendous amount of value out of unit testing. But SQLite is quite abnormal in many respects. It's used in aviation, so the standards of testing are ridiculously high. It's also a library, and libraries tend to be very easy to test. It's also a long-term project with no deadline to speak of.
The way I read the article it sounded like he was downplaying the usefulness of even healthy amount of testing i.e. writing tests along with the code your writing.
Certainly, all his arguments downplay TDD in general
"If you're a Java programmer and want to have a rude awakening, go download Jester. Jester is an automated mutation testing tool - it goes in and replaces "<=" with "<", "&&" with "||", "!" with whitespace. And then it re-runs the tests. And then you get to watch in horror as your tests still all pass, regardless of what the product code actually does."