Yes. Agree about TDD being more suited to low dependency software like CRUD apps or self contained libraries.
Also sometimes even if the individual parts aren’t right, the whole can still work.
Consider a function that handles all cases except for one that is rare, and testing for that case is expensive.
The overall system however can be written to provide mitigations upon composing — eg each individual function does a sanity check on its inputs. The individual function itself might be wrong (incomplete) but in the larger system, it is inconsequential.
Test effort is not a 1:1. Sometimes the test can be many times as complicated to write and maintain as the function being tested because it has to generate all the corner cases (and has to regenerate them if anything changes upstream). If you’re testing a function in the middle of a very complex data pipeline, you have regenerate all the artifacts upstream.
Whereas sometimes an untested function can be written in such a way where it is inherently correct from first principles. An extreme analogy would be the Collatz conjecture. If you start by first writing the test, you’d be writing an almost infinite corpus of tests — on the flip side, writing the Collatz function is extremely simple and correct up to large finite number.
Computer code is an inherently brittle thing, and the smallest errors tend to cascade into system crashes. Showstopper bugs are generated from off-by-one errors, incorrect operation around minimum and maximum values, a missing semicolon or comma, etc.
And doing sanity check on function inputs addresses only a small proportion of bugs.
I don't know what kind of programming you do, but the idea that a wrong function becomes inconsequential in a larger system... I feel like that just never happens unless the function was redundant and unnecessary in the first place. A wrong function brings down the larger system feels like the only kind of programming I've ever seen.
Physical unit tolerances don't seem like a useful analogy in programming at all. At best, maybe in sysops regarding provisioning, caches, API limits, etc. But not for code.
> I don't know what kind of programming you do, but the idea that a wrong function becomes inconsequential in a larger system... I feel like that just never happens unless the function was redundant and unnecessary in the first place. A wrong function brings down the larger system feels like the only kind of programming I've ever seen.
I think we’re talking extremes here. An egregiously wrong function can bring down a system if it’s wrong in just the right ways and it’s a critical dependency.
But if you look at most code bases, many have untested corner cases (which they’re likely not handling) but the code base keeps chugging along.
Many codebases are probably doing something wrong today (hence GitHub issues). But to catastrophize that seems hyperbolic to me. Most software with mistakes still work. Many GitHub issues aren’t resolved but the program still runs. Good designs have redundancy and resilience.
A counter to that could be all the little issues found by fuzz testing legacy systems and static analysis. Often in widely used software where those issues did not indeed manifest. Unit tests also don't prove correctness, they're as good as the writer of the unit test's ability to predict failure.
I can tell you that most (customer) issues in the software I work on are systemic issues, the database fails (widely used OSS) can corrupt under certain scenarios. They can be races, behaviour under failure modes, lack of correctness on some higher order (e.g. having half failed operations), the system not implementing the intent of the user. I would say very rarely those are issues that would have been caught by unit testing. Now integration testing and stress testing will uncover a lot of those. This is a large scale distributed system.
Now sometimes after the fact a unit test can somehow be created to reproduce the specific failure, possibly at great effort. That's not really something that useful at this point. You wouldn't write that in advance for every possible failure scenario (infinite).
All that said, sometimes there's attacks on systems that relate to some corner cases errors, which is a problem. Static analysis and fuzzers are IMO more useful tools in this realm as well. Also I think I'm hearing "dynamic/interepreted" language there (missing semicolons???). Those might need more unit testing to make up for the lack of compiler checks/warnings/type safety for sure.
The other point that's often missed is the drag that "bad" tests add to a project. Since it's so hard to write good tests when you mandate testing you end up with a pile of garbage that makes it harder to make progress. Other factors are the additional hit you take maintaining your tests.
Basically choosing the right kind of tests, at the right level, is judgement. You use the right tool for the right job. I rarely use TDD but I have used it in cases where the problem can relatively easily be stated in terms of tests and it helps me get quick feedback on my code.
EDIT: Also as another extreme thought ;) some software out there could be working because some function isn't behaving as expected. There's lots of C code out there that uses things that are technically UB but do actually have some guarantee under some precise circumstances (but idea but what can you do). In this case the unit test would pass despite the code being incorrect.
I work in software testing, and I've seen this many times actually. Small bugs that I notice because I'm actually reading the code, which became inconsequential because that code path is never used anymore or the result is now discarded, or any of a number of things that change the execution environment of that piece of code.
If anything I'm wondering the same question about you. If you find it so inconceivable that a bug is hiding in working code that is held up because the calling environment around it, than you must not have worked with big or even moderately sized codebases at all.
> sometimes even if the individual parts aren’t right, the whole can still work.
And in fact, fault tolerance with the assumption that all of it's parts are unreliable and will fail quickly makes for more fault tolerant systems.
The _processes and attitude_ that cause many individual parts to be incorrect will also cause the overall system to be crap. There's a definite correlation, but that correlation isn't about any specific part.
Yes. Though my point is not that we should aim for a shaky foundation, but that if one is a craftsman one ought to know where to make trade offs to allow some parts of the code to be shaky with no consequences. This ability to understand how to trade off perfection for time — when appropriate — is what distinguishes senior from junior developers. The idea of ~100% correct code base is an ideal — it’s achieved only rarely on very mature code bases (eg TeX, SQLite).
Code is ultimately organic, and experienced developers know where the code needs be 100% and where the code can flex if needed. People have this idea that code is like mathematics where if one part fails, every part fails. To me if that is so, the design too tight and brittle and will not ship on time. But well designed code is more like an organism that has resilience to variation.
Also sometimes even if the individual parts aren’t right, the whole can still work.
Consider a function that handles all cases except for one that is rare, and testing for that case is expensive.
The overall system however can be written to provide mitigations upon composing — eg each individual function does a sanity check on its inputs. The individual function itself might be wrong (incomplete) but in the larger system, it is inconsequential.
Test effort is not a 1:1. Sometimes the test can be many times as complicated to write and maintain as the function being tested because it has to generate all the corner cases (and has to regenerate them if anything changes upstream). If you’re testing a function in the middle of a very complex data pipeline, you have regenerate all the artifacts upstream.
Whereas sometimes an untested function can be written in such a way where it is inherently correct from first principles. An extreme analogy would be the Collatz conjecture. If you start by first writing the test, you’d be writing an almost infinite corpus of tests — on the flip side, writing the Collatz function is extremely simple and correct up to large finite number.