coverage reports. They tell you which branch wasnt executed in your test, which can tell you what may be missing in your unit testing.
> more turning to functional regression testing when entire service is tested as a black box against spec.
functional/regression testing is important, but it can be difficult to use if your app is huge. And you can not easily isolate an area to test, so you always end up testing a common component repeatedly, and thus is slower to run.
Not to mention some functional testing mechanisms (such as using a web browser) can be flaky.
Both have their place in the automated testing paradigm.
Although coverage reports are misleading, 100% coverage means that you passed through every line of code, that's it.
You don't know if you covered edge cases, it cannot cover concurrency well and although possible it will not tell you whether you used a series of values or long-paths in your code.
Yes. And it's worth noting what's left out of that. Passing through every line of code is not the same thing as following all paths through the code.
Here's a sampling of code features where that distinction is particularly important:
- successive if-statements
- switch statements with fallthrough
- loops
- mutable variables with a large influence
The first two are because {{a}, {b}} ≠ {{a, b}}.
By that last one, I mean both mutable variables with a large scope, and mutable variables within objects that have a large scope. That's probably the trickiest thing, because, in the presence of mutability like that, it's not just which lines of code were executed that matters, it's also the order in which they were executed.
That doesn't strike me as a pretty picture. Overall, it implies that automated code coverage metrics are the least useful in the kinds of codebases where they are most desirable.
You're right about edge cases in terms of input values, and right about issues with concurrency, but many of the issues you mention are solved by changing your metric.
Branch-level coverage tells you what percentage of codepaths are covered.
In other words, it would report 50% coverage for:
if a:
print "a yes"
else:
print "a no"
if b:
print "b yes"
else:
print "b no"
test 1: a = false, b = false
test 2: a = true, b = true
Real coverage test would test coverage OF THE SPECIFICATION. So for a list of statements describing how your service should behave in various situation you would want to know which are being verified by the unit test.
Checking lines of code is pointless because line of code would usually be required for multiple specification statements (requirements).
There is barely any connection between having 100% line coverage and covering entire component specification.
I think unit coverage are those measures where somebody thought they need to measure something but they could either not define what to measure or they deciding that measuring the right thing is too difficult so we can measure something else instead and managers will be happy anyway.
coverage reports. They tell you which branch wasnt executed in your test, which can tell you what may be missing in your unit testing.
> more turning to functional regression testing when entire service is tested as a black box against spec.
functional/regression testing is important, but it can be difficult to use if your app is huge. And you can not easily isolate an area to test, so you always end up testing a common component repeatedly, and thus is slower to run.
Not to mention some functional testing mechanisms (such as using a web browser) can be flaky.
Both have their place in the automated testing paradigm.