In fact it only gets in your way. Solving problems that are new (to you) involves writing code to find out if a certain approach works, then throwing it away and trying something else. TDD makes that process a lot slower, and makes people more reluctant to throw away incorrect models of the problem that they have codified into tests without even realising it.
Of course somebody is going to chip in here and say that if the above is happening then you are writing your tests at the wrong level of abstraction and that you should be testing at the point where you can create a stable API that remains unchanged while you throw away the speculative implementations underneath. But if you are tacking a new problem you have no idea where those boundaries lie. It's hard enough to determine that for problems that you already understand and have working solutions for, never mind ones you don't. TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all. Either that or you just write vast vast numbers of trivial tests for every single function that you write (which does seem to be what some TDD tutorials advocate), but then you are creating a huge maintenance burden for the future where any change to the code is going to necessitate rewriting dozens of tests.
I would argue that TDD only works if you are solving problems that you are very familiar with. By the time you are writing your 5th CRUD web app you probably have decent mental model of the parts you need before you start. I feel that a lot of the love for TDD comes from people who spend most of their time working on problems they already solved dozens of times before, but they learned how to solve them before they discovered TDD.
For people first learning how to code it is absolute death. It discourages people from doing the most important thing they need to learn: throwing code away.
> TDD won't help you solve problems that you don't already know how to solve:
I talk a lot about web application security. The following line from flesh and blood developers on multiple occasions:
"Shouldn't your tests catch security vulnerabilities?"
To which I always respond:
"Yeah sure, they'll catch the vulnerabilities that you already thought of"
I've personally seen codebases with test suites that cost (even with the most conservative estimates of time spent and day rate) upwards of 300,000 USD with 100% coverage that allowed an anonymous client to drop the users table.
If a lack of security bugs (and therefore bugs in general) is any indicator of software quality, TDD has yet to impress me in the wild.
My development process has always been to prototype out three or four implementations, evaluate them against the known use cases and then settle on a final API design. Then I'm ready to do TDD. Glad to know I'm not the only one who works roughly this way.
It also bugs me a little that many otherwise intelligent developers use TDD as their sole software design tool much of the time. The exact term they use for this is using tests to "drive out" a design. While I understand this may work in textbook examples, in real life I've seen the concept being used to justify designs that clearly aren't fit for purpose.
That illustrates a very serious problem with TDD, the idea of coverage.
100% code coverage is not 100% coverage of every different possible code path. In fact, for most software systems it is not even remotely feasible to achieve that level of code path coverage. The "100%" figure can easily lull less experienced devs into thinking they have sufficient test coverage when nothing could be further from the truth.
However, TDD encourages – ideally – a style of programming where 100% branch coverage is likely to mean the program is correct.
That's part of the idea of the "unit" in "unit test." You decompose the program into units for which correctness is relatively simple – and testable.
How to do this properly is not obvious, and for teams on which TDD is imposed without this understanding, it will all become a bizarre and useless ideology.
When it is understood, the resulting program will have properties that resemble those of purely functional programs, especially composability and correctness.
If you were to ask anyone in my office, you'd hear how much I praise TDD as a software development practice. However, when I'm absolutely clueless as to how to even begin to tackle a technical problem (usually due to not enough understanding of the overall system requirements), then I start by spiking the problem - trying to get a feel as to everything that's required. This initial spiking I wouldn't test-drive, as it's a learning process and not production-ready code.
After I've got a strong enough grasp as to the overall requirements, then I throw away this spiked code to start fresh, working along similar lines to 'programming by wishful thinking' whilst keeping the original spiked out structure in mind.
By following a strong TDD philosophy, it's easier for me to throw away my initial 'spiked out' code base for a couple of reasons:
1. Personal experience of how much better a test-driven code base can be than ones I've later tried to wrap tests around/dismantled in the process.
2. Having the understanding from the get-go that this code is not 'production-ready', and should not be considered reliable, but as a very fast learning experience.
3. Knowing that I've spent a small time prototyping as preparation, and having this laid out as an overall part of the process means I'm not considering this time 'wasted'.
When tackling a new problem, you still need to build a lot of utility functions that you know how to write. Tests help here.
For example, I'm building a statistical model. I need to load a bunch of data from files, transform it in straightforward ways, then run my model against it. TDD doesn't help me come up with the right model, but it does help me not screw up while loading the files. When my model fails to be predictive, I know the model (rather than the loader) is at fault.
Absolutely! In the past I worked on an image recognition project where I implemented, tried and thrown away 14 distinct algorithms/approaches (not counting variants!) for a particular problem. TDD is unfit for this because you tend to commit yourself to the set of tests instead of throwing all of them out (all of the approaches were fundamentally different and I had soft real requirements -- so optimized representations were a must). Of course we had some unit tests, but not written in TDD style since the goal was to find the correct approach -- not the correct implementation.
Not 14 different things. 14 different approaches to solve the same image recognition problem. Please do not jump to conclusions. I did not say TDD is not useful at a _later_ stage, but at the research phase it is simply more of a burden than help, because the "refactor" step ends up as "drop everything".
> TDD won't help you solve problems that you don't already know how to solve
So, those Sudoku testing article fails because the author is breaking rules of TDD. I mean, the first test: test_cells, does so much more than it needs to do to pass the test. He's designing first, and then testing. That's not how TDD works.
If you can't see why that first test is a bad test, you aren't qualified to judge the merits of TDD.
> Of course somebody is going to chip in here and say … you should be testing at the point where you can create a stable API that remains unchanged
That's not true. At least, not the way you phrase it. You are, of course, forgetting that TDD involves 3 steps. Writing a test, Passing that test, and then refactoring. That refactoring includes refactoring the test. API's can, and should change to evolve. That's part of refactoring.
You pretty much repeat the same arguments against TDD that I see time and time again. Arguments born out of misunderstandings. TDD isn't easy. It's challenging, it requires practice and effort. It does take time.
> TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all.
Yes, but nothing about TDD prevents you from easily making changes as needed. Again, Refactoring is apart of TDD. You say something like this multiple times, making it seem like refactoring or changing code is some painful process.
It's not. It's trivial. If it's not, your process has a serious problem.
if you are looking at Ron Jeffries Doyen of the Agile set, and saying "well of course he did not get the results, he was doing it wrong" the we are well into id-protecting, religious territory.
if one of the leading exponents gets the process wrong there is something wrong with applying the process to the real world
TDD really seems to me to be two parts:
- Design up front, disguised as writing an API
- Ensuring you write repeatable tests before commuting
> if one of the leading exponents gets the process wrong there is something wrong with applying the process to the real world
I'm more interested in extracting what works from a concept than slavishly following a cult leader. IME, being a "leading exponent [sic]" of anything doesn't make you infallible, even in that domain.
> Design up front, disguised as writing an API
The TDD literature I am familiar with doesn't advocate writing an API up front, it recommends writing essentially a minimal fact statement up front about the API, writing tests to verify that fact statement, writing code to pass that test if it doesn't already pass with the existing code base, and iterating like that. You could design more than the single fact that you make the test and code for at one time, but that's not a demand of TDD or even relevant to it, and is not, in most cases, consistent with the kind of decide-as-late-as-possible rule that the Agile and Lean approaches in which TDD is usually embedded prefer. I suppose it would make some sense if you were doing waterfall-adapted-to-incorporate-TDD, but why on earth would you do that?
I didn't check with the literature, but in my opinion, the biggest misunderstanding of TDD is that lots of people take it to mean literary Test-First Design.
As I understanding, in Test-DRIVEN-Design, you are writing the implementation code in PARALLEL to testing code. You write a tiny bit of test, then a tiny bit of implementation to satisfy the test. If you notice something you'd missed previously, you refactor BOTH tests and implementation. You repeat and expand a bit...
Read the literature. Kent Beck, the author of the idea and the book describing it, very clearly argues for test first. The test tells you how to design. I'm not sure if that is what you mean by your paragraph or not. But in his book, the test is always written first, and it guides you into how you should write your code.
I found it extremely dubious. He was doing some kind of accounting/banking app, and came up with a new approach. Well, he spent his career on this domain. I find it far less likely that TDD caused the new idea than it was just that he'd hoed this row so many times before. If you have done it 10 times before you can do bottom up, top down, inside out, outside in, or completely random, and get a reasonably structured, working code base. I conclude nothing by him creating accounting app #11 with a scheme that is perhaps well suited to how his brain works, but is perhaps terrible for mine.
I'm not against testing, quite the contrary, but I find some of the claims rather exuberant.
> Read the literature. Kent Beck, the author of the idea and the book describing it, very clearly argues for test first.
No, you are misunderstanding the parent. He meant some people assume you write _all_ tests first. If you read his entire comment, you'll see how he describes exactly the process described by Kent Beck. A small test, a small implementation, followed by refactoring both test and implementation (the 3rd part being the part commonly forgotten).
> You write a tiny bit of test, then a tiny bit of implementation to satisfy the test. If you notice something you'd missed previously, you refactor BOTH tests and implementation.
Bingo! Exactly that. Right on the money, especially the part about refactoring both tests and implementation.
If you didn't check the literature and have your own understanding of what TDD is, then it should be no surprise that what you are talking about is different from what everyone else is talking about.
> You write a tiny bit of test, then a tiny bit of implementation to satisfy the test. If you notice something you'd missed previously, you refactor BOTH tests and implementation.
That's TDD.
He was alluding to the fact that people might assume that they have to write all the tests first.
She/he is talking about exactly what everyone else is talking about. That's TDD. "Didn't check the literature" may have meant that the reply didn't come after reading the referenced book.
Yes, if you can't solve a problem, you certainly can't solve a problem in a supportable manner.
Agile has the concept of Spikes (http://www.scaledagileframework.com/spikes/) for researching new problems. The key point that code is thrown away after completion -- this gives you the freedom to ignore normal methodologies and not have to worry about technical debt.
Presumably, you then go back with this new knowledge and only now concern yourself with writing supportable code.
From my experience, I've found that TDD is mostly valuable where writing the test will make writing the code more efficient. A lot of the times its because manually testing it to get the code right will add significant amount of time and then you will still need to unit test anyways. If I don't know exactly what I need to do, sometimes I will write the code to get a proof-of-concept working then just comment it out. Then I write the tests and uncomment the code as the tests are written to make sure I have proper coverage and I haven't missed anything.
I hate writing tests that require me to do any kind of boilerplate code. Tests should (theoretically) only set up and exercise whatever is needed for that test. I think if you are able to do that (I've had success with AutoFixture for .NET as an example) then you'll feel a lot less bad about throwing away tests and they will also be much much faster to write. When writing tests is tedious or brittle, then I feel they are a chore and am less likely to write them to begin with.
sometimes you can't even really express what the problem is, until you've taken a stab a solving it. The problem might be something like 'This feature is awkward and doesn't feel quite right.' When I'm doing exploratory coding, I need to be able to feel my way through a problem. I have yet to learn to do that from the vantage of tests, and am somewhat skeptical that I ever will.
You can't really express it, but you still have an idea of what the problem is. . ."solving it". . .you have an "it" in your mind. Also, in many cases, when you are working on a "less technical" feature, there are others who can help express what the problem is. . .user, designer, architect, etc. It's often really tough for a developer to express what the problem is in a vacuum.
I was taught that you should solve the problem first, and then write the code. TDD doesn't change that; it forces you to at least define the problem before you take a stab at writing the code.
Of course, as you delve deeper into the problem, you often learn something that changes your understanding of the problem. That's where the third step of TDD comes in - refactor. TDD doesn't make people reluctant to refactor their code; it forces them to do it. Refactoring is a vital part of TDD.
TDD does allow you scope for exploration work. If you watch Kent Beck's TDD screencasts[1] you'll see him follow a workflow where he's exploring an unfamiliar protocol. What's interesting is that he first frames his exploration within a particular test e.g. delete item using protocol, and then writes implementation details inside the test itself until he figures out how it works. He then refactors the test so that the implementation code is moved into the driver that he's developing and the test itself becomes a proper spec. He's working in java but I tend to follow this workflow in ruby. With ruby you can fire up a REPL in the middle of your test code and gradually add to the test as you figure things out.
>In fact it only gets in your way. Solving problems that are new (to you) involves writing code to find out if a certain approach works, then throwing it away and trying something else. TDD makes that process a lot slower, and makes people more reluctant to throw away incorrect models of the problem that they have codified into tests without even realising it.
This is a giant strawman. Who is saying to write tests for your prototypes? You seem to be confusing a bunch of different methodologies here. You're talking about rapid prototyping to understand a domain before building a long term solution. That isn't TDD, and it would be foolish to write tests for an exploratory exercise that you're planning to throw away.
>TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all
How do you figure? If you've done a rapid prototype like you're advocating, you should have a better understanding of where boundaries lie and how the API should work. Further, you seem to be working under the impression having tests make it hard to change things. If anything, I've noticed the opposite - tests make it much easier to change code, especially when you test behavior, not implementation.
If you are iterating their isn't always distinct prototypes — just quick first-draft implementations. Sometimes these are good enough.
In my experience once everyone is happy with the way something works then it is sensible to move to lock-down — tests, documentation, performance testing, refactoring.
Doing any of those four things is a waste of time and energy until the design is decided upon. In agile development the design is not decided upon before the first code is written. It is always jointly discovered through iteration and discussion between designers, user-testers and developers.
If you write your documentation first then I suspect you can get away with writing all your tests first. But that's not a process I would call 'agile' and I doubt it ever results in good software.
I interviewed with a company one time that ran with TDD and pairing like a religion.
The guy I was interviewing with didn't like the fact that I thought about how the problem might be shaped (and therefore made assumptions as to what I might want to implement and refine via tests) prior to “writing” that first failing test. It was absolute nonsense and I was rather shocked that the company managed to ship anything, to be honest.
I'm a big fan of tests; I'm a big fan of pairing. They are tools; nothing more.
An interview recently felt like that. I was just throwing together little bits of code to make sure I understood the problem clearly, but the interviewer seemed to be a bit agitated that I didn't write a test right away.
I honestly don't work like that. So I'm kinda glad I failed that interview.
Yeah. I've recently done a couple of interviews on the other side of the fence (as the interviewer), and I've hit on a problem that can be worked out collaboratively between the interviewee and the interviewer—and how far we go depends on their skill.
It always begins a discussion of the problem and what we want to solve, plus a bit of how—so we can figure out what it is that we're going to be testing for and how to build from that point.
I think the point is more the the final "prototype" is the "implementation". The core fallacy of TDD is that there is a single moment where you start work. True iterative development doesn't work that way.
Which is not to say that both TDD and iteration don't have their uses. But they work together poorly. And the problems best solved by TDD seem boring to some of us.
TDD is supposed to work WITH iteration. You have one feature, you write the test for it, you implement it, pass the test, and go on to the next feature, etc.
Implementing features piecewise is not remotely the same thing as iterative development. The point to iteration is that you develop your understanding of the problem along with the solutions, many of which get thrown away.
It seems like there are a sea of absolute assertions with few facts backing them up - people seem to love either saying that TDD is better or TDD is worse, and it seems that the only data ever really offered on the subject is anecdotal opinion.
We can do better than this. This study[1] at Microsoft reveals that TDD results in a defect reduction rate of between 40% and 90%, at a cost of 15%-35% increased development time. So, what's the best approach? It now becomes a business question. If speed of delivery is crucial and quality less so, don't use TDD. If quality is more important that speed, use TDD.
And finally, everyone involved in building software owes it to themselves to take an hour to go and watch this excellent video by Greg Wilson entitled "What We Actually Know About Software Development, and Why We Believe It's True" [2], and then perhaps dive in to the list of references from it [3], because you will find yourself more actually informed than anecdote and opinion will ever get you.
As the other poster pointed out, this paper is measuring nothing, or at least it is not written in a way to let us know.
I find it unremarkable that a sudden focus on quality and testing reduced the bug rate. It does not follow that TDD is the cause, nor that TDD is the best of multiple options.
Its a well known effect in experimental design - measurement alters what you measure. Get depressed people to watch and talk about a dog video. I can predict that they will be less depressed. Not so much because dog videos are great at reducing depression, but because all of a sudden so much experimental interest is being directed at them. It would be almost churlish to not feel better, if you know what I mean. So, we don't design experiments that way - if your hypothesis is that dog videos cure depression, you need to do exactly the same protocol, but with cat videos, or I don't know, truck videos, or whatever, so both groups get the same attention, both are trying to reduce their depression and so on (I'm assuming unblinded here because obviously the TDD paper was unblinded, obviously we have better methods for the depression study than my suggestion here).
Kudos for bring empiricism into the discussion, but I do find the paper lacking.
I wrote above that I don't use TDD yet achieve low defect rates and good design. I think it is because, no matter how you do it, if you focus on those things you will more likely achieve them than if you don't. I hypothesize (don't be mad!) that TDD works for some simply because it brings the issue of quality to the forefront of their mind. Whereas, I'm already always thinking "is this line of code going to kill somebody"; I've tried TDD, and it always seemed like it added little to actively inhibited me. Your mileage will vary.
I've been trying to find empirical information about software development and he's frank about the lack of it. He noted work by Lutz Prechelt [http://www.inf.fu-berlin.de/w/SE/OurPublications], for one.
TDD is often praised for its documentation effect, vital for teams working on big projects with heavy moving pieces. But I've found TDD to be essential in projects in which I'm the main or sole coder. OftenI can't finish every piece of a project in a single day, so I'm switching context and code bases quite frequently. TDD has been the only way I can keep on task, knowing exactly where I left off by running the suite. It also not only forces me to write more modular code, but it abets the reuse of code because if a module works in one project, I can extract it and put it in the others and can use TDD to immediately see if that raises issues.
Even for side projects, TDD has been essential...it's so easy to let side projects die especially if you've forgotten where you left off. With some TDD, I can discipline myself to at least write a few tests today if I can't muster the energy to do real code for the project...and often that little push ends up with lots of code being written....just as you can go on a five mile run as soon as you motivate yourself to get off the couch
TDD is like any other tool. Abuse it and you'll get terrible results, use it well and it will save you time and effort over the lifespan of a project.
The big tricks are: test modules on interface boundaries, tests should test one thing and one thing only, don't overdo it and realize that tests are code and will need some maintenance.
Then when refactoring time rolls around you'll be a happy person.
If you overdo it or test in other places than at the interface level you'll find out that with every change half your tests will fail and that maintaining the tests is more work than maintaining the code. Tests will be 'brittle' and will test code at a level that is not useful.
I'm tired of people asserting that TDD results in better X, while simultaneously stating that TDD does not guarantee good X. What does that even mean? It sounds like a non-falsifiable statement, that aims to blame the person whenever your methodology fails.
Another issues is that no one ever qualifies what exactly is TDD better than? Better than someone vomiting hundreds of lines of code into a single file with no systematic approach? Better than test-last development? Literate programming? Better than Design by Contract, advanced type systems and static verification?
In blogs, TDD proponents often completely ignore the difference between unit tests and integration tests, pretending that there is none, and that TDD will automagically produce both, even when their methodology does not provide any guidance for designing integration test. (The article does this too: "The test suite, that you grow along the way, will help get those features implemented, and bugs fixed, without breaking other features." Uh, not necessarily.)
And what I'm really, really tired of is the assertion that TDD produces not just more reliable, but better-designed and otherwise better code. What I've seen in real life is that TDD encourages developers to bulldoze through the problem by constantly spitting out code and tests without bothering to think at the higher level. That results in a lot of "enterprise Java" style abstractions and fractured code that is extremely hard to read and debug. (Yes, it's easy to understand what every particular unit does, but it's not at all clear how they interact and why we need all of them.)
Here is a simple example of deficiencies of TDD that uses the infamous prime factor Kata: http://insideofthebox.tumblr.com/post/52002125683/prime-fact...
There is a great question in that article: which solution would you rather work with if you had to modify it?
One small problem I have with TDD advocacy (as distinct from just TDD) is that it can make it seem like an all-or-nothing proposition.
Either you start all projects and do TDD all the way or you eschew automated testing altogether. And the former is the "right" way to do it and the latter the "wrong" way, ignoring that in practice a little bit of testing goes a long way.
Imagine this. You have just been asked to work on a project with a very short deadline, on a codebase you aren't that familiar with, in a language you aren't that keen on. You are capable with it, but you aren't a big fan. You are going to have to go and add features to this already reasonably large codebase and it has no tests at all because it was hacked together.
What do you do to ensure you don't break it? If it is a web app, write yourself a simple set of automated integration tests that use a ghostdriver-style API like PhantomJS. There's WATIR in Ruby, there's Splinter in Python etc. etc.
You then have some emergency guard rails in place. They won't be the best guard rails. They won't be as good as if you'd been doing testing from day one. But they help you get your head around the application in the large while you are tweaking away with stuff in the small. And because it's not your code, you need some guarantee that it's not breaking.
TDD advocacy is fine but a bit too perfectionist. In the real world, there's lots of untested code. Don't make the perfect the enemy of the good. Apply the Pareto principle: write the simplest test code you can to cover 80% of the existing features. Prioritise by importance ("how many people would the client want to murder if this went wrong when being tested by the end user?"), don't let theology get in the way of you getting shit done.
I was really hoping for some insight here but in the end it was just another TDD post that amounted to nothing more than handwaving a feel-good statements. I mean it wasn't even backed up logically forget empirically. This amounts to a thesis without any supporting premise.
I've written software both with and without heavy TDD. TDD does not result in higher code quality, being knowledgable about coding does (in fact amongst more junior developers TDD results in both bad code and bad tests which is doubly costly). This is TDDs dirty little secret, not that it can take longer.
Don't get me wrong tests and test automation have tremendous value as a tool. But TDD as a processes for building all software is frankly just silly and if you want to convince me otherwise you're going to need real evidence.
It should be pointed out that what's actually being discussed is whether TDD is better than not writing tests. What's not being discussed is whether, if you commit to writing tests, there are more effective ways of doing it.
Yes, this is the fallacy that TDD advocates use every time (even after it gets pointed out to them repeatedly). They pretend that anybody who doesn't use TDD isn't writing tests at all. It's incredibly frustrating.
Almost every point for or against TDD in this thread is based on factors that go beyond TDD itself.
Many seem to have a bad taste in their mouths from experiencing bad tests, for whatever reason. Often this is a result of programmers who have yet to learn how to write effective, flexible tests. Proper TDD, just like everything in software development, requires experience, dilegence and patience in thinking through the problem space.
Developers who are for testing, at least in my experience, tend appreciate the side effects of writing tests; simplified interface code, code base stability on evolving projects and flexibility to change implementation without breaking features.
That being said, I agree that for new problem areas, TDD is often a burden. If you're approaching somethin new (a new tool, protocol or external api) it makes a lot of sense to write throw away code until your understanding of the problem domain evolves. At that point, when the problem is understood enough and the feature is to be implemented into a production system, it is appropriate to write a solution using TDD.
Ultimately everyone has their preferences and, depending on the problem and situation at hand, it's up to the developer to decide if the long term uncertainty of a code base without TDD is better than a short term gain in productivity. Both approaches are appropriate, given context.
If you definition is loose, then you could probably say that. Generally though, using a TDD approach would result in some form of a test suite that can be used to continually assert the correctness of a program.
Personally I attempt to write tests whenever possible, even in a new problem domain. However, I generally only write some general "black box" style tests to start.
There are several claims on here, that I could address by replying, but I'll try to top post. I see several claims about how much better the code is with TDD. Well, can I see examples? I write tests, but I don't do TDD. By and large, I am proud of my code; I think you could design things differently, but rarely better. I spent 17 years doing this for the military, more for cancer research, where results and bugs matter, and I strongly believe (absent proof, I didn't do it with TDD after all) that my solutions were near optimal in terms of resources used. Meaning, how long it took me to make the bug free code, how understandable and modifiable the code was, and how defect free it was.
So, where is the code that will blow my socks off? May I see some? I recognize that is a difficult challenge to answer explicitly; I'm not really asking for a github link or whatever. I can't show you mine, for one thing, and neither of us will be able to accurately measure who did it with fewer resources. So, hypothetically, in words, show me your better code.
I'm not trying to blow your socks off—I think the typical programmer will say "yuck!" when looking at any code they haven't seen before—but I thought you might be interested in the examples.
Both of these codebases were developed "live" on camera. You can see the screencasts here:
So, at my old job, we had 'TDD', but we weren't particularly good about it. Maybe it's because Python's testing frameworks aren't as worked out as Ruby, maybe it was because we were lazy. I genuinely don't know, I wasn't trying hard at the philosophy.
But when we finally got to work at my new job, our lead made TDD a requirement. Every time. Red. Green. Refactor. I still screw it up sometimes(as he'll attest), but God, it's like a relevation.
Probably the nicest part, is simply being able to just know every time that you don't need to be scared of a bad git merge. Scared of hunting down a bug that was made 3 weeks ago but no one noticed till now. Scared of deploys.
All of that is mindless for us. Pull down master, merge it in, run the tests, be happy. Pull Request, code review, merge to master, let Travis-CI handle the deploy.
It helps keep code more simple, makes the API more useful, and to me, I don't know that I can go back. The moment you're past week 2 or 3 of building with it, it's invaluable.
What were you using for python tests. I used nose on a project[1] and it seemed to do everything I needed.
Your second sentiment is the exact same thing that happened to me. I got that revelation moment where I'm like "holy crap" we made changes and merged everything and it's actually working. Which, sadly, wasn't the case before we went into the TDD model. I mean even if the test cases were added after the code was written, I still noticed a lot faster development cycle and a lot fewer bugs on the projects that were using TDD.
TDD was a step we took to learn how to create testable code that was decoupled and had a single responsibility.
It also taught us that we needed automated user level regression tests.
But, TDD is like training wheels. After you have gotten those things from it, it does start to hold you back.
I think testing is very much something that follows a Pareto curve. You want to be at the right place on the curve, or else you are either missing a lot of relatively easy coverage, or you are spending a lot of time and money for small gains in coverage.
On a tangent, one of my current projects involves a hardware system that interacts with people. That makes testing things much harder than a simple Rails app where there is an abundance of helpful tools to use. Any ideas?
I was looking into this: http://quviq.com/index.html - and it looks interesting. It's quite expensive, but it might be worth it if it can significantly increase and automate test coverage.
I'm really rather surprised that we're still talking about "is TDD worth it?" To me, it's a bit like asking, "is OO..." or "is functional..." or what have you. I should hope any developer I work with would know TDD and would know when to use it and when not to use it.
TDD is a style of development that emphasizes simplicity of code interface and accuracy and precession of results. However, not all project have these three things as requirements. In those areas where one has a distinct model of their problem, TDD excels and will save you tons of time over a long run of forcing you to consider corner cases and avoiding breaking existing code with new changes. It's excruciatingly difficult to test things in the fuzzy logic and statistical realms, as you always run the risk that your artificially generated data set does not reflect reality very well.
And none of this should be news. This should be standard information about TDD. It should be clear that TDD is a tool that makes sense when TDD makes sense. Just as it doesn't work well to write an OO abstraction of general-purpose sorting algorithms, and it doesn't work well to write a purely functional simulation of a car, so does it not work well to TDD certain things.
I have committed professional suicide on more than one occasion by voicing my doubts about the whole TDD and worshipping at the alter of "unit testing school".
TDD works when problem is well defined but complicated e.g. a datetime library. TDD is a tool to get the grunt work out of way and take care of occasional forgetfullness/regression, but beyond that it is a pile of code that becomes more and more unmaintainable with each passing day. I have seen unit testing code which grows into a snowball over a period of time and people wasting their time to maintain tests that they don't know a thing about. IMHO you shouldn't write a single line of code if you can get away with it e.g. if you need to write unit tests for your add(int i, int j) then I guess you have bigger problems to solve.
I always take comfort in something a colleague said long time ago while coming to my rescue in one such outburst "are there any unit tests for *nix, vi, emacs etc, the kind of software I dont see crash very often"
Disclaimer: My TDD worshipping friends seem to do better in their professional lives, which I will shamelessly put down to industry trends/religion rather than the true merits of the methodology ;-) It suits me better...
I think I've had the same experiences and made the same observations as you. Also, +1 to this: "...you shouldn't write a single line of code if you can get away with it."
My problem with TDD is that it, fundamentally, isn't agile. It either relies on a waterfall-like process where the full design is known before-hand and therefore what tests have to be written, or it locks a programmer (and designer) into a position where to iterate and/or backtrack would require not only new code to be written but new tests.
It it is very hard for a designer to ask for changes once tests are written.
I much prefer a looser arrangement where code (and design) are gradually locked down after prototyping, backtracking and iteration. Code gets thrown away and rewritten but tests are only created once — after the implementation (and by definition the design) is locked-down. It is the very generation of the tests that tells the world that this design is unlikely to change and documentation is safe to create, performance tests & optimisation passes are able to be made.
Of course, after tests are created, a programmer is free to refactor safely and replace all the code as long as the tests continue to pass.
There are occasions where it makes good sense though. For instance, if you are writing a Maths library or the core business logic of an accounting app. However, these are both essentially examples of a waterfall-like development process.
It's not true to say that TDD isn't agile. They are orthoganal - you can be agile and do TDD and you can do waterfall development and not do TDD. The fact that you need to change tests to change the software is the whole point - if you change the functionality, you change the tests. In fact, it's not wanting to change the tests that causes the problem here.
Perhaps you have been on bad projects where TDD has been used improperly. However every single agile development team I have worked with has used TDD, and it certainly does not limit things in the way you suggest.
Ironically, Extreme Programming [1], one of the first of the agile methodologies, and many other agile methodologies since, include TDD as a core tenet. So in fact by saying "TDD isn't agile" you are perhaps demonstrating a misunderstanding of what "agile" development was designed to be.
In fact, it's not wanting to change the tests that causes the problem here
This is the point I was making. Programmers never want to change the tests as, often, it has taken as long to create the tests than it did to write the code. It adds a layer of syrup to the whole process. You can no longer be nimble as you are carrying the weight of the tests.
And, yes, I've worked with precious designers who don't want to change their design when a first iteration clearly shows that it isn't working.
But to be truly agile (and I'm not talking about some written methodology here — I mean agile as in nimble) you need to work quickly, try things out and make mistakes. Writing your tests first is a encumbrance.
Perhaps you have been on bad projects where TDD has been used improperly
This is what everyone says the moment their dogma is questioned.
I've read the books, I've been on the training courses and I have certificates in my drawer.
My experience is, if you want to create anything well-designed you need to iterate and there is no point in writing tests until your design is fixed it just slows everything down.
> But to be truly agile (and I'm not talking about some written methodology here — I mean agile as in nimble) you need to work quickly, try things out and make mistakes. Writing your tests first is a encumbrance.
Perhaps it makes sense to be explicit about this when talking about it, as "agile" has a commonly accepted meaning in software development terms and this is not it.
> This is what everyone says the moment their dogma is questioned.
You have said "TDD is that it, fundamentally, isn't agile" and have asserted that you can't use it to be agile. I have stated (in a sibling comment) that TDD is a business decision, where you trade increased development time for higher software quality, and cited a study on the matter that is non-anecdotal. Your position is the dogmatic one by the commonly accepted definition of "dogmatic".
OK lets examine the commonly accepted meaning of Agile Development and see where TDD fits in.
TDD is at odds with half of the Agile Manifesto, namely:
* Individuals and interactions over processes and tools
* Responding to change over following a plan
You should first try stuff out and discuss it rather than working out how to machine test it before its been tried by a human. Having tests first is clearly having a plan first.
TDD also rubs up against two principals of the Agile Manifesto, namely:
* Welcome changing requirements, even late in development
* Regular adaptation to changing circumstances
It's hard-enough to welcome changing requirements when you have to rewrite software without having to rewrite all the tests too. Why write tests for code you don't keep?
Changing and adapting is harder when you are encumbered by tests. Tests should be written once you've agreed that what you have now is what you are going to keep.
TDD adherents I have worked with in the past also rub up against this one:
* Working software is the principal measure of progress
Tests are not 'working software', they are a protective covering you put over working software to make sure it doesn't get ruined as people work around it.
At its heart Agile is about being adaptive: evolutionary changing plans, iteration, rapid and flexible response to change.
TDD brings overhead: tests that needn't ever have been written; tests that needn't ever have been re-written. They hamper iteration and flexible response as they make it more costly to change plans.
Perhaps you haven't seen it work, I have seen it work many times, and like I said, it is about tradeoffs. You trade increased development time for higher quality software.
I think it's quite bizarre to be arguing that TDD is not agile when it is an explicitly agile methodology. Kent Beck, the inventor of TDD, is an author of the agile manifesto that you quote. The authors of the document clearly disagree with your interpretation of it.
>You should first try stuff out and discuss it rather than working out how to machine test it before its been tried by a human. Having tests first is clearly having a plan first.
This is a strawman argument. TDD does not imply working out how to machine test something before it's been discussed and planned by a human. If you need to try it agile has a concept of spikes. This is not an argument against TDD because this does not describe TDD.
>It's hard-enough to welcome changing requirements when you have to rewrite software without having to rewrite all the tests too. Why write tests for code you don't keep?
Tests are part of the code. Why write code you don't keep? Yes, you have to throw away tests if you throw away code but this is not a TDD issue. The issue is you're throwing away the code (of which the tests are a part).
> Changing and adapting is harder when you are encumbered by tests. Tests should be written once you've agreed that what you have now is what you are going to keep.
No, changing and adapting is easier when you have tests because you can be sure that you haven't broken anything. If tests fail when you change things, either the requirements have changed so you change the tests, or you've accidentally broken something and the tests have caught it. Tests should be written at the same time as the code to validate that the assumptions of the developer at the time the code is written remain true.
> Working software is the principal measure of progress
The working part is important - if comprehensive tests reduce defects in software - which is a pretty uncontroversial statement, if you disagree with this statement you are at odds with all the published research on the matter I have seen - then writing tests helps your progress by helping to deliver working software.
> TDD brings overhead: tests that needn't ever have been written; tests that needn't ever have been re-written. They hamper iteration and flexible response as they make it more costly to change plans.
And again, I repeat, this is an engineering tradeoff. Is speed more important that quality for your business case? Then TDD is a bad fit.
What designer? The front end guy? The PM? What are you talking about? Have you got an architect making edicts from high?
That in itself sounds like it is fundamentally not agile.
I'm not especially a fan of TDD, but this comment makes no sense to me, it doesn't marry with my experience creating software.
TDD is around individual methods, trying to make them decoupled. It is around unit tests. Changing the design simply means deleting those methods and throwing away those unit tests.
Small agile teams don't have a designer and a 'front-end guy' and a 'PM' and an 'architect'. They often only have one and in my case he (I) is also the PM (or PO in our case).
Design-led software requires a designer. In large projects you might break this up into lots of different roles with trendy titles but they are all just designers, just as a database programmer and a graphics programmer are just programmers.
Changing the design simply means deleting those methods and throwing away those unit tests.
Which is wasted work, wasted time and totally dispiriting for a developer.
> It either relies on a waterfall-like process where the full design is known before-hand and therefore what tests have to be written,
This is that fuzzy line between behavior driven development and test driven development. Sure, you may not be able to write cukes to test how that your social mortgage site is going to work, but you probably should be writing tests to ensure that your internal compound interest code is correct.
I say exactly that in my comment. I'm not suggesting that tests shouldn't be written — I'm suggesting that tests shouldn't be re-written without just cause.
Yes. It helps you code much faster. Tests run soooo much better than the brittle look-and-poke method of testing your implementation.
Also much more reliable.
TDD might not be worth it if you're working in a strictly typed language with no side-effects, or it might be a bit less worth it. But in a dynamic or non-typed language you have to do that half of the compiler's work in tests. Especially if there's more than one person touching your code.
I think it depends on your development environment. If you're writing something small and it's just you, you'll understand everything that you're doing (hopefully). If you're building a complex system with multiple parties and you're practicing separation and rotation of duties (no one owns any piece of code), then YES. I've gotten to the point that even for interviews where I'm asked to write code, I'll add test cases into whatever I'm building.
I've worked on projects where code isn't tested and I've made a simple change thinking that's all I had to do to fix some "bug" only to find out by the QA team that I've broken three other features. Features which test cases would have been able to spot at build/test time. This took an hour fix and turned it into a week long back and forth with QA and the original developer.
On the other side, I just wrote an iOS game and it has 0 test cases. I pretty much know what's going on, I'm the only developer on the project and I don't see it getting much bigger (famous last words right?).
I've been a TDD practitioner for a while now and I've lately started to question the value of doing the refactor step within the red/green cycle, even if the code to make the tests green is a horrible kludge. Sure it's going to add technical debt to your project, but I can't shake the feeling like it is usually premature optimization of the maintainability of the code. Especially if it is a new feature, the requirements of maintainability seem like they're probably pretty low at first since you could find out that the feature was completely unnecessary in the first place for example.
In a talk once, I heard an offhand remark that a good signal that a method needs refactored is if you're frequently fixing bugs in it because it signals you don't have a good grasp on what the method is supposed to be doing, and I've started trying to use that rule in side projects to optimize more for getting things done inthe few hours on nights and weekends I have to work on them.
I've found a good approach to resolving the last issue is to write the documentation for the method. This forces you to think about what you want it to do outside of the implementation.
Right, but if you're never going to look at the method again and/or remove it next week for whatever reason, you've just wasted a bunch of time that could be spent on more valuable things.
My thinking is definitely more focused on early stage start ups/personal side projects than bigger mature projects.
But what I'm trying to get across is that I'm starting to wonder if "big clunky code base that's hard to support, maintain and understand" is in the same class of good to have problems for early stage start ups/side projects as "I chose a technology that is tough to scale to 500 million users" as long as you have a comprehensive test suite.
I tend to vary the amount of refactoring I do depending on how much code is likely to depend on it. For instance I'd spend more effort in the DOM than I would in the controller (remember this is degrees of effort not lots/none). There's also nothing to stop you doing refactoring _before_ you start on a piece of code. Maybe you didn't think it would be worth refactoring at the time but now you're adding a new feature you can see that applying the refactoring would make it easier.
I was very keen on using TDD to lay out the design and technical specification of a solution to be implemented by our team (8 developers total).
Unfortunately, because the project has tight deadlines (regulatory project), despite minimal requirement clarity the team needed to get going and in the end every developer ended up with just a rough guideline of how to implement the various components.
Does anybody have ideas for integrating TDD with an Agile approach and a team around this size?
I've done Extreme Programming (cf. _Extreme Programming Explained_ et al) approaches on a team of about that size and it worked okay, but the deadlines weren't as hard. The pair-programming aspects helped keep everyone informed about what sort of implementation decisions and tools were being made (quite useful on a team of eight) without bogging things down with meetings, and it also helped keep people very focused... and also honest about actually being TDD, as in "don't write a single line of code which isn't necessary to make some test pass", which helps keep you from overengineering some part of the code and helps you meet deadlines.
But it's hard to say whether it'd work, because (a) your problem is described very vaguely, and (b) your team might not be up for it (the classic problem), and (c) even if they were up for it, it's a process they likely haven't practiced much, so that makes it harder
Like many said, it also bothers me that I don't believe that TDD code is any better than non TDD code with identical unit test code (or path) coverage.
The fact that when this blog post says TDD, it means "writing unit tests" is really annoying. TDD is a methodology of writing features by writing tests first to define the behavior you're targeting, then writing code to match that behavior. That is different than "write code, then write tests to assert the behavior of your code." You can have excellent tests and excellent code coverage and excellent behavior documentation without doing TDD.
In fact it only gets in your way. Solving problems that are new (to you) involves writing code to find out if a certain approach works, then throwing it away and trying something else. TDD makes that process a lot slower, and makes people more reluctant to throw away incorrect models of the problem that they have codified into tests without even realising it.
Of course somebody is going to chip in here and say that if the above is happening then you are writing your tests at the wrong level of abstraction and that you should be testing at the point where you can create a stable API that remains unchanged while you throw away the speculative implementations underneath. But if you are tacking a new problem you have no idea where those boundaries lie. It's hard enough to determine that for problems that you already understand and have working solutions for, never mind ones you don't. TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all. Either that or you just write vast vast numbers of trivial tests for every single function that you write (which does seem to be what some TDD tutorials advocate), but then you are creating a huge maintenance burden for the future where any change to the code is going to necessitate rewriting dozens of tests.
I would argue that TDD only works if you are solving problems that you are very familiar with. By the time you are writing your 5th CRUD web app you probably have decent mental model of the parts you need before you start. I feel that a lot of the love for TDD comes from people who spend most of their time working on problems they already solved dozens of times before, but they learned how to solve them before they discovered TDD.
For people first learning how to code it is absolute death. It discourages people from doing the most important thing they need to learn: throwing code away.