The whole technical debt discussion often makes me wonder if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.
Do people in other professions have the same issues? If the salespeople say this thing must be done in this particular way, or if the lawyers say this thing must be done in this particular way, are they questioned and overridden by their managers? I honestly don't know; it feels like software engineers are given a special level of distrust but it just might be that I'm perceiving that because I haven't experienced the other jobs.
FWIW where I currently work, we have an amazing middle-manager / product owner who has to fight the same battles with upper management with regards to process improvement and technical debt. She gets it, but has to fight the same battles we do. Sales and C-level want everything now with as little investment as possible.... I wonder if I just answered my own question?
I distrust my fellow software engineers when they say they need time to pay tech debt.
There were many times where I refactored code that never received any additional updates or to have the feature removed. I wasted a lot of time without any real gain for it.
There are real costs to paying off tech debt and it is easy for engineers to spend their time on low value refactoring.
Software engineers are not always good at prioritizing the business value side of their work so they aren't trusted there.
If you can show you can prioritize business value then it is easier to convince management to give you two weeks to clean up tech debt.
> There were many times where I refactored code that never received any additional updates or to have the feature removed.
Since we're throwing out anecdotes, I've lost count of the number of companies I've seen that were essentially zombies due to technical debt. They had aging, rotting codebases that were impossible to add features to at the rate needed to continue making sales. No senior engineers who could improve the situation wanted to set their careers back by working for them.
The main problem is that many developers will increase technical debt when they try to fix it. For example, some will try to fix technical debt by writing a new framework. Then the new framework doesn't handle all edge cases, so you still need the old solution. Now you have two different conflicting styles creating more technical debt. The next developer will come in, see the mess, and promptly start working on a framework to consolidate everything...
So whenever you consider fixing technical debt, ask yourself if the team is skilled enough to actually create a cleaner solution. If they can't then it is just a waste of time. The best predictor of this is that they have created a cleaner solution elsewhere, if they haven't then don't let them work on technical debt unless they have an extremely well thought out plan.
Management's solution is often to hire more software engineers.
Ironically in big corporations I think that this actually plays into their underlying desires - giving a boost to headcount gives them more importance and having more developers means that the leverage each engineer has is commensurately less.
I've worked on more than a few projects which would have proceeded 4x faster with 2 good developers on a clean code base than the 7 they actually had.
Indeed the entire software startup industry is probably reliant upon this dynamic.
Not the software startup industry, unless you're flush in easy VC cash. Most of the startups which are trying to solve real problems that I've known/interviewed for are extremely conservative with hiring, for the very good reason that bad hires are not just a 0 value add, then can be a negative value add.
Uber is one of the outliers in this regard. But if you look at most successful others (Instagram, Whatsapp and a bunch of others I don't want to name since I want to stay anonymous) all had very small teams right until they were acquired.
I meant that the startup industry would be hollowed out if large corps didn't exhibit this dysfunction and thus create space in the market for them, not that startups do it themselves (although occasionally they do too).
My experience is that these companies have the worst kind of technical debt: a prematurely optimised and poorly abstracted codebase, due to non senior engineers refactoring just in case. Non abstracted codebases are boring to work with because the same code is duplicated, but they are not that difficult to change or add feature to. Extreme DRY on the other hand are the kind of codebase everyone is afraid to touch. Sure maybe they had 100% test coverage in their days, too bad adding a new feature with a small model change would break every interwined object factory.
It's not just extremely DRY stuff that can be problematic.
I work in a non-abstracted code base, but all that really means is every time someone wanted to do something they simply wrote their own way of doing it. Needless to say it's suboptimal to work in on the best of days.
People like to deride one way of doing things vs another but in reality pretty much every approach to software development and architecture can be amazing or hellish depending on the aggregate quality of the developers, their management and the lines of communication between them.
TL;DR - Bad code is bad no matter the paradigm and all paradigms can be made awful.
I think you and grandparent post's author are both right. I've seen both. It depends on the type of engineers that inhabit your team, and therefore are not mutually exclusive.
> Software engineers are not always good at prioritizing the business value side of their work so they aren't trusted there.
This.
Software engineers are focused on the implementation side of things and how today's work will impact their work in the future. Yet, sometimes a quick and dirty solution that takes close to nothing to implement is more than enough to put a functionality in production. Sure, the whole thing will need to be scrapped in the near future when rewriting the feature to meet all design requirements, naturally the quick and dirty solution underperforms, and of course implementing it properly will take 3 or 5x the time. Isn't it obvious that the team is wasting it's time by implementing a quick and dirty solution?
Yet, the quick and dirty feature does shorten the time to market to a fraction of what it would otherwise take to implement properly, and perhaps that is good enough for the project for the time being.
It's for this reason that I've often tried to get the business side to give opinions on how shoddy/quickly they'd like the feature to be done. Curiously a lot of the time they appear very reluctant to do so. It's probably CYA kicking in maybe... many managers don't like to say "do a shoddy job" if it ends up biting them in the ass later... they'd often rather just hint and exert unspoken pressure to rush it out and then deny everything later when the CEO starts asking questions a bug that fucked everything up.
There are exceptions, of course, and I'm pretty happy to have worked with them, but in general about 90% of managers I've worked with have resisted my attempts to get them to be explicit about the level of quality that they want.
My take is that product managers care mostly about what functionalities were delivered and if the requirements are met. If the development teams deliver a quick and dirty solution and it meets some requirements then they check the requirement as delivered, and that's it.
When this happens then the devteam's negotiating position changes completely, because they are no longer arguing about how to allocate resources to deliver a functionality and instead they are asking to waste time rewriting something that was was already delivered and is supposed to be up and running.
Quality is so ridiculously subjective that I'm not sure a non-technical manager should approach the question, and a technical manager should refuse a direct answer and steer the conversation towards structure and deliverables.
% of time spent on refactoring is completely objective. Whether that results in quality is another matter, but it's a dial that a product manager has control over that (usually) has direct impact on the quality of the end product.
"Continuous attention to technical excellence and good design enhances agility."
Very few people seem to actually believe this in their bones. All talk about being "agile" and "pragmatic" becomes a joke when it's posed as somehow being in contradiction to design and anything that goes deeper than finding a place to insert your new if statement.
> If you can show you can prioritize business value then it is easier to convince management
If your management needs 'convincing' then there's a problem.
Either the manager is technical enough to know whether a particular set of work is the right thing to do or not, in which case they don't need convincing. Or they're not, in which case they should defer to the engineer and also not need convincing.
The only managers who need 'convincing' on technical matters are technically inept individuals who nonetheless attempt to micromanage technical decision making. They are no more able to link business value to technical decisions than the engineers you say you distrust. Fire them.
> Engineers don't generally enjoy paying off technical debt, it's a chore. But they recognise that it must be done sometimes.
I disagree. Most engineers I work with like refactoring. It’s usually well defined work with clear metrics for success. It can also reduce daily pain if it makes delicate/messy code much easier to work with.
The problem with refactoring is that its only value is in improved execution of future projects. So it can be extremely valuable or completely useless. It depends on how likely the refactored things are to be worked on again.
It’s because there are literally infinite ways to solve similar problems in software. There is no physical assets to measure the differences making very hard as relatively speaking it’s still a very young domain. I still go back to my freshmen year in college when in my dorm we had a mechanic, chemical and material science engineers. Each of them complained because they were not senior enough they could not get time in the expensive lab to run their end of year projects... it was at that moment they I looked at my crappy dell laptop and realized I had access to the most advanced systems in my field computer science and it was just sitting here in my dorm... I mean a few years later a guy on similar crappy laptop or desktop created Facebook... and yet there are so many options to build Facebook using php, c#, c++, Java, python etc... maybe to you one of those ways is inferior or less good then the other. Maybe using cloud servers or physical would be better?? Maybe a class or a function... css or Scss... the space is infinite
I think it’s about perception. Broadly speaking, professions like Doctors and Lawyers are seen as prestigious and important, and so they are allowed to manage themselves.
I think software engineers tend to overestimate how prestigious and valuable people outside of tech think software engineers are.
I work in computational biology, and even in a fairly tech related field, you wouldn’t believe how many people think software engineering is just glorified IT. I’ve had a lot of conversations where the crux of it is “why should we pay them so much money, it’s the computers that are doing the work. Or, on building a distributed computation engine to run for 2 years straight without failing: “it’s all just grunt work we’ll just get an undergrad to do it”.
> why should we pay them so much money, it’s the computers that are doing the work.
Why should we pay the blacksmith so much money? It's the armor doing all the work. Why should we pay the stone mason so much money? It's the arch that's doing all the work.
Yes that’s essentially the problem. Engineers/developers aren’t seen as fundamentally making anything. They’re just seen as the interface to the computer.
> The whole technical debt discussion often makes me wonder if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.
Yeah.
I think, sadly, this is mostly on software developers though. Look at all the pop culture stereotypes of programmers: whether those stereotypes are true or not, that is what culture thinks of us. Would you take seriously what a big-bang theory character would say? The problem is, I don't see most software engineers valuing social skills to better relate to other professions. Instead, there's a lot of whining about being misunderstood. And we are misunderstood! But man we're not helping it at all.
Contrary to most stereotypes, most adult software engineers are fairly well functioning adults. They may not be the best at public speaking or other social skills, but they're not mostly like characters from the Big Bang Theory.
I agree, but if the split is 90% well functioning, and 10% people with the manners of RMS/Torvalds, well, people are going to notice those 10%. Stereotypes obviously don't need to be real to shape things. My concern more is that our industry needs to stick up for itself and act like professionals, instead of intentionally marginalizing itself and then acting indignant about it.
I don't even think that 10% really needs to change. We need the RMS' and Torvalds of the world. But we also need to, individually, act better than they do and advocate for ourselves as adults.
I constantly see coders deriding any decision that places the business over their own individual interests, and then we get indignant that the decision makers don't listen to us. But if we don't care about their interests, and the collective best interest of the company, why should they care about our interests?
From my experience, managers tend to start to distrusts software engineers after the first complete system rewrite, that was supposed to solve the big problems, actually didn’t solve them and brought new ones instead.
The issue is neither has crossed the bridge. The non-technical manager can’t understand the risks, and the engineers can’t quantity it in the dollars and cents that CFOs understand. Salespeople are good at dollars and cents. So are some lawyers. But they get overruled all the time by execs.
“I don’t care how you do it, increase sales by 20% in Q4” creates the sales and marketing version of tech debt. (Channel stuffing, excess discounting, etc)
I'm glad you've raised this, because it's something I've felt for a long time.
A few friends of mine work in accredited professions, and it's not something they have to deal with all that much, most likely because they are all accredited. They are billed by the hour, and typically if someone isn't happy with the cost the response is "okay, we'll down tools". Inversely, companies that bid on contracts will bid with a fixed cost in mind, and when estimates are taken that go over the requirements things will be fudged to fit the requirement in for that price. This means one of three things: corners will be cut, developers have to work faster, or requirements are cut. In my experience, the last point is a rarity.
I know that we software engineers like to think of ourselves as thought workers, and on the same level as the likes of lawyers and chartered accountants, but the way our industries operate are wildly different. I prefer to compare software engineering to a trade like carpentry, plumbing, etc. We share many similarities, such as easy barrier to entry and working on fixed-cost projects, but unlike trades, we're always willing to negotiate on our next pay day. You can say it's down to a difference in costs, but even on an industrial level where the person doing the sale isn't a trades-person, if a job requires £x and the client can only pay £y, then the requirements change or the job isn't taken on.
People in other professions have similar issues. Some 'smart' folks feel as if they should be able to understand everything. It probably often works for them, after all, how did they become 'smart'? But, it's a fine line between genuine interest and distrust per se. Try to assume good intent, consider it interest and grow your ability to explain. At some point you might have to call it as a poor use of time and move on, but if you can use the time to grow on both sides of the discussion - bonus.
This is what I suspected, with my perception being tainted by the "grass is always greener".
I also can't say we aren't collectively to blame, or at least somewhat - as a bunch we can be poor at political negotiation/skill, lack of standardization, etc. I kick myself every time I miss an estimate that I've given, because I know at some level that has to erode trust. Or when my manager does press for more details as to why something should be done and I can't come up with a really strong justification at that moment, etc.
> if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.
Physicians enjoy a similar problem. Everyone goes to Dr. Google an then brings their misunderstandings to the clinic. Anything goes wrong, it couldn't possibly have been the fact that the patient was a brittle diabetic with hypertension. It must be the care they received in the hospital after they had a heart attack while driving and ran into a tree. Nor is anyone else in the hospital possibly making mistakes. Clearly it's the board-certified physician who doesn't know what they're doing.
Off-topic, but the ‘Dr Google’ thing is interesting. Something that makes it more nuanced in that particular case is that a patient will have better insight into the state of their own body in many ways than a physician will.
I’ve been watching a series called ‘Mystery Diagnosis’ about people with rare diseases and how they were eventually diagnosed. Something you hear most of them say is along the lines of ‘Finally I met this doctor that actually listened to everything I had to say’. It’s a really interesting series — and damn does it remind you how lucky you are to be mostly healthy!
Yes, business stakeholders often mistrust software and IT people, because relative to other industries we do a poor job of meeting our commitments.
McKinsey did a study recently which found that large IT projects ran on average 45 percent over budget and delivered 56 percent less value than expected [1].
An executive is interacting with people from multiple industries or business functions and this is on the high side. It's much worse than a function like accounting, and it's also worse than many industries with bad reputations, like construction. So over time they are going to come to regard estimates from the tech guys with more suspicion than most. To the extent that other industries have this problem, yes you will find similar trust issues.
If you are a PM/product manager then the best way to handle technical debt is to focus on the benefits that come from paying it down. You are not scheduling time for technical debt, you are scheduling time to improve reliability, security, scalability, etc. and from day one you are lobbying for more time to do every feature so that it's done right in the first place. Yes, sometimes one of these things has to go wrong before the business people start listening, but if you were calling it out from day one they will respect your input that much more.
If you are a developer it's important to understand what work has business value and what doesn't, and make or inform decisions appropriately. Frankly there are developers out there whose attitude amounts to "Your job is to pay my salary, my job is to make whatever technical decisions I want without oversight, and if we don't deliver on time/on budget it's not my problem."
Not all tech debt is worth paying down, the decision generally should be made based on the "interest rate" of the debt (how hard is this going to burn us in the long run?) and the term length of the "loan" (will this component be around for one year or ten?).
This approach fails spectacularly. The benefits of paying down tech debt are often massive but they are extremely diffuse (i.e. they tend to affect everything in a small way) and hard to recognize (good luck proving that feature X that hasn't even been dreamed up yet will take 16 days instead of 25 because of Y refactoring you're doing now).
Every time I've ever seen the approach of "create a business case for each refactoring" put into practice it has made the problem spectacularly worse because the developers are reluctant to manufacture sales pitches for things where the payoff is hard to recognize.
Hell, developers are reluctant to manufacture sales pitches period.
They're typically awful salespeople. If they're good at making business cases, and few are, they're probably not working for you. Making the quality of your product contingent on the skills of your developers at creating sales pitches is a brain dead own goal.
The best approach is to hire good engineers, allocate a rough proportion of their time that proxies the level of quality you want (e.g. 30%) and trust them - let them figure out how to allocate that time. Coz if you can't trust them you're probably fucked anyway.
>If you are a developer it's important to understand what work has business value and what doesn't, and make or inform decisions appropriately. Frankly there are developers out there whose attitude amounts to "Your job is to pay my salary, my job is to make whatever technical decisions I want without oversight, and if we don't deliver on time/on budget it's not my problem."
I can't begin to explain how incredibly damaging this attitude is. It's the philosophy of a micromanager.
Seriously, anybody who wants technical oversight of developer decisions should either become a developer or they should fuck off.
> McKinsey did a study recently which found that large IT projects ran on average 45 percent over budget and delivered 56 percent less value than expected [1].
Is that fault of IT people or rather did business people over-promised? Because my repeated experience is that we give estimates, they are considered big and shorter estimates are sold. And it has nothing to do with IT people over-promising. I have yet to see a team where it is tech people pushing for shorter deadline and management being like "yeah, it is too risky, lets go with bigger estimate".
There are situations of me making short estimate, but I also make estimates too big in other situations. However, somehow the estimate that makes its way toward contract is always short due to negotiating pressure.
PM are all too often non-technical, it is quite unfair to blame "IT people" on mistakes of general management and project management. Project management blaming the these contracts being basically frauds on anyone else then themselves is literally them blaming scapegoats for their own faults.
I think one of the problems is that code is a black box to non-programmers. If I tell you that your car needs $2000 worth of work in order to avoid a $10K problem down the line, how do you evaluate that? Keep in mind that as far as you can tell the car is performing exactly as you expect at the moment.
Another problem is that especially in large organizations there are people who are literally trying to scam their way into money. You always have people aggrandizing their work, or belittling others. Especially as companies become profitable, they attract people who are only out for their cut. There are people who do nothing but attend meetings and try to scoop money into their pockets. This is the norm.
Now marry the two issues and you can see the result: someone says they have to do "special work" that will delay the project and that they are actually saving everybody else's asses. Nobody that isn't involved in doing the work can evaluate the claim. Are they just avoiding work/blame? How do you know?
I find that usually in organisations I work with, there is trust up to the D-level manager (i.e. those who directly manager developers), but there is often very little trust from other parts of the organisation (sales, upper management, or sometimes even program managers).
The best way to engender trust is to work as tranparently as possible. In the best of worlds you would deliver something every day and it would be obvious what's in the queue. Things would be in the queue for a very short time (say 2 weeks max). Work like refactoring, writing tests, improving build systems, etc would be just things you do every day and nobody would ever mention it. Of course, not everybody lives with an ideal situation, but I think most teams should strive to move their situation as close to this as they can (there are definite exceptions, though).
In a sense, technical debt is only a big problem for large monolithic codebases. Software that is primarily composed of small manageable components means that technical debt doesn't matter as much.
When each component is roughly one week's worth of code, it becomes easy to delete a component with a model that no longer fits the problem domain, and rewrite a new component with a new more appropriate solution.
"If you can rewrite a piece of code in a week, you can understand it in a week." - Greg Young
Except when you large monolithic codebase is composed of a whole bunch of screwy distributed components that have been thrown together haphazardly. And, yes, I know you can't have a "monolithic codebase" with "distributed components". It's just that it's actually all the same dish with a different sauce. Distribution of processing doesn't create better code bases -- it's decoupling of the computation that does it. You can decouple code and a very large number of ways. It's also just as easy to write a rat's nest of confusing spaghetti with components that hold each other's state as it is to do it without "components" (I speak from experience on this front, as much as I hate to admit it).
Well if components are holding each others states then that's failing the decoupling test.
But I agree, it's definitely not easy to keep things properly decoupled. Requires some level of discipline or some savvy choices up front in the overall architecture of the whole app..
We distribute senior titles to people with three years of experience. We task junior developers with setting up and managing critical cloud infrastructure. Management has no idea how it all works.
And we often hire people who have no clue how things work even for low-level management positions. Can you imagine in construction, the lowest level manager being someone with no clue about construction but is just supposedly really good at "project management"? No way that would ever happen. The "management has no idea how it all works" issue is real but it's not because the CEO doesn't know how it works. The CEO doesn't know how things work in any industry.
In SD we have people with 3 people under them who are clueless. There is no construction company in the world where a guy who has 3 people under him doesn't understand the actual work being done.
No wonder there is a trust issue when there are such huge knowledge gaps, it's not uncommon that someone's first direct superior already understands very little about what they do.
Although I have a feeling that this issue is not present at "top" companies. I have a suspicion that at a company like Microsoft, you'd have to go quite a few levels up the chain before you get to someone who is technically clueless. I doubt they'd hire some guy who never wrote a line of code and have him lead the Visual Studio team or whatever. This is something shitty companies do, it's not an unavoidable fact of life.
Basically there should never be a situation where someone's job is to write code, but their boss can't understand that code. The knowledge gap between boss and employee is too large. A boss always needs to be able to understand the work their employee is doing. If you're at a higher management level where your direct reports do not write code, then sure, you don't need to understand code.
I think it's a bit different in software because it's fairly easy to change things and update them.
As for the distrust, generally there is can be a lot of distrust in things you don't fully understand. Legal I think is a great example. There are a lot of contracts that screw people over and legalese is so hard to read, I have a hard time trusting what is going on. But one big difference is it is very difficult to go back and change legal terms. I was changing an LLC to a c-corp in order to receive investment and it took 3 months to fix the "legal debt" I had incurred by starting with an LLC with outside parties who had a say in what the new terms were. And it's hard to say whether we did all the right moves in that transition because I'm not a legal expert. Just as some people doubt the decisions I make in software.
I think this is related to how badly things go wrong if there is a fuckup. You really do not want to push the engineers responsible for determining your $500M office building will stay standing.
So what kind of fuckup in software is going to have to happen before this is also true? My fear is that it will happen one day, if not already (see eg- 737 MAX). I'm not looking forward to that day.
> Do people in other professions have the same issues?
Yes, any domain which doesn't produce a product, conceivable to a layman in one way or another. You need conceivable results to be understood by managers and other people. For example domains producing tangible results are doing much better.
I used to be electronic engineer, and it was way simpler to explain laymen around you that you actually doing something. Blueprints, boards are conceivable more or less. Lines of code are not, especially hard to explain that less lines of code is better, since laymen don't see architecture, they see just text.
There is a widespread misunderstanding of what Technical Debt actually is and why it comes about. It's not about poor coding standards, it's a rational pursuit of commercial advantage through deferral of costs.
I've written a few words about it a while ago here: https://datalanguage.com/features/reassessing-technical-debt
It might have something to do with (1) how expensive it is to make software and (2) the fact that software engineers are known to sometimes be purists or aesthetes or susceptible to hype.
A good comparison might be (building) architects: you know that if you let an architect run wild, there's a good chance you'll end up with something that is beautiful but wildly impractical and over budget.
(And to a lesser extent, every expert wants to do a good job, not a good enough job.)
> The whole technical debt discussion often makes me wonder if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.
LOL, seriously? Guess you haven't worked that much with other practitioners, for example lawyers.
There is some distrust always with business relationships, and I would say that it makes sense not to trust everyone 100% all the time. In the opposite, better not to trust everyone fully, even yourself.
Here's what normal people see and hear when you enter the room: Nerd opens mouth: "Blablabla quibberish, you're all idiots, hear me express your problem way overcomplicated. Also, I need more budget and cannot promise to actually deliver a solution anytime soon because of more quibberish."
If you want to be trusted start acting like normal people. Stop talking in technical quibberish. Stop behaving like a nerd. Shape up. Get some social interaction with normal people. Stop complaining all day long and start delivering what was asked on time and on budget.
I love the metaphor of technical debt - you take it on with your eyes open, and sometimes you chose to pay it down.
But... in larger engineering organizations I've found that "paying off technical debt" isn't actually a particularly useful goal - because teams can use it as an excuse to work on pretty much anything!
If a team spends three months rewriting some code into a different style that they prefer, is that a useful pay down of debt?
So - I prefer more targeted debt repayment. The best approach I've seen so far is to focus on "get rid of duplicate systems". If you have two systems that solve a particular problem and you knock that down to one, you're guaranteed to have improved things - and it's very easy to explain to non-technical stakeholders why having one system is better than having to maintain two.
> ... because teams can use it as an excuse to work on pretty much anything!
This is a great observation that explains the uncertainty ive had from time to time.
Literally anything can be considered technical debt to someone! It's extremely vague subjective and is usually a bad reason to change/rewrite the code. Better reasons include; untestable code, scaling problems (now not predicted) and of course duplication...
I tend to agree, as I've seen it used and misused from both sides already. While the higher echelon often misses empathy for technical debt, what they need rather than a "euphemism treadmill" is real and actionable education on the _cost_ of technical debt.
Likewise, engineers do tend to refactor everything and their mother when left alone with too much time and with too little education on opportunity cost and context on the big picture.
The best advice I can give is first creating awareness and secondly making nonfunctional requirements part of any product negotiation. Always quantify the implicit expectation "...and it works": How long, for how many people tops, with how much possible downtime,...
More often than not, business stakeholders can get away with much less than engineers think. That way, lots of technical debt and bad examples of overengineering never come up.
Collect a few things that could be done way better / faster / stabler, ... — and then just convince everyone that a better architecture is the best way to get there.
> The best approach I've seen so far is to focus on "get rid of duplicate systems"
In my experience this still requires finding the right balance, like almost anything else in programming. If you're too eager to get rid of duplication you can end up combining things that aren't actually quite the same and creating leaky abstractions that are a lot more painful than the almost-duplication was.
There's also sometimes a trade-off involved. Maintaining two very similar systems can be more work/require more engineers, but maintaining one system that is used in different ways can make you more likely to introduce bugs unexpectedly if you make a change focused on one particular use without testing all the others enough. It might depend on the particular system which cost is more acceptable.
Yeah I've definitely seen people refactor code to eliminate duplication in a way that turns out to be counter-productive a few weeks later, when it turns out the two places that use the code are evolving in different directions.
I'm thinking more about duplicate systems. A really common pattern in a codebase that has been around for a few years is incomplete replacements: someone builds v2 of the checkout flow, but it never quite completely reaches feature parity with v1 for all of the edge cases so both of them stuck around. Kill those!
Yes it's important to keep separate things that could be seen as duplicate but the meaning of which is different and will evolve differently. By example paying user and delivery user or provisioning an IP address and a physical server.
Oh. That’s helpful! Makes it much easier if you ever have to modify the behavior of addition! Like, if a new discovery in math is made and it turns out that 1+1 isn’t always 2.
The real benefits will appear at testing, where you can replace the actual addition with a mock object that expects parameters 1 and 1 and returns a fixed value 2, without having to do the calculation.
My approach is to highlight technical pain points during the week and at the end of the week vote on the biggest pain point and then allocate a fixed amount of time to work to fix that in the next week.
Usually there are about 10-12, 7 of which can be safely ignored and 1-2 which are critical.
"pain points" is a good expression here. Some code inflicts more pain on the developer than other and the psychological effects should not be underestimated (=reduction of productivity due to lower morale).
I also like your approach of time-boxing the fixes.
> The best approach I've seen so far is to focus on "get rid of duplicate systems"
In our case (very much performance-oriented), we're actually moving toward duplicating systems, because the cost of a centralized system (in term of latency and communication overhead) has become unbearable.
> I don't see why the analogy to debt is necessary when the direct analogy is to capital underinvestment: in fact what we're talking about is or very nearly is just that, capital underinvestment, not simply something analogous to it. And the hypothetical businessy, non-technical types whom the debt analogy is intended to enlighten understand very clearly what deferring capital investment is and what the likely consequences are, at least if they actually know anything much about business.
> Most of the people who are pushing for "features now, someone else's problem later" are probably not upper management but rather middle management. Upper management might understand capital underinvestment. Capital underinvestment is very nearly the sole job middle management is there for. And berating people for not being done yet. At least at a couple of the places I've worked.
That's because you've just made the term up. If you meant instead "the underinvestment problem", then that's actually the opposite -- that's avoidance of pursuing new features because it'll incur technical debt.
Coming up with better terms for "tech debt" seems like a good exercise. Here are some:
Improving the flexibility of our code base.
Investing in future execution speed.
Performing final cleanup tasks on the previous round of features.
Also, when a report comes and asks for time for "refactoring," I ask them what they want to improve. Having a particular goal or target area for the refactor makes it more successful, and has the side benefit of making it easier to communicate the benefits to the rest of the company.
I agree, the narrative is important. Cleaning up "technical debt" may sound like a big cost and delayed eng schedule with no clear benefit for non-eng teams, despite being crucial to day to day operations.
As it should, honestly. What's really happening is that, usually due to deadlines, a job is left unfinished, and the undone tasks (like cleanup, optimization) are then labeled as "debt" just in order to push a ticket do "Done" earlier.
I have noticed that business folks roll their eyes when "tech debt" comes up. I sympathize, as it didn't used to be that way. COBOL on a mainframe would remain current for decades. Tools and languages are very fragmented these days. I recently tried to get a popular node.js app (tech radar) up and running, but npm just kept bombing out on abandoned packages. The project is 2 years old and already "debt ridden".
Funny, because it's all client side/ browser JavaScript. Node isn't really needed, but that's the toolchain. So it's borked for no good reason.
This makes me think part of the problem with tech debt is that it means different things to different people. Some people might describe it as a codebase just not keeping up with this month's latest and greatest JS frameworks, but I think of it as something much more nefarious- low level shittiness and incoherency that tends to seep into long-lived large projects over time. Eventually the lack of refactoring to account for new features or behaviors turns it into a giant blob of spaghetti, where everything knows about everything and what should be minor, localized changes can have unforeseen consequences. This is harder to deal with IMO because it's something that appears so slowly that it's hard to stop in isolated instances ("WTF, you wanna rewrite this whole class instead of just commenting out that one line and adding one more tiny code path to fix this bug right now?!?"). That type of technical debt is more of a boiling frog problem where it's not obvious to everyone how bad it is until serious consequences have started to arise.
Maybe instead of "paying off tech debt" we should sell it as "frog-cooling." I'm only half joking. They'll respond to that by saying "WTF?" and you can be like "Yeah the frog's been in the pot not realizing it's heating up, and is about to boil." Gives it some urgency.
In general I don't think tech debt should be communicated to the business people.
It's an internal part of engineering work, and doesn't impact users. You can think of this as "encapsulation" if you like.
Also what it sounds like to outsiders is "we're taking time off to fix things we didn't bother to do right in the first place". And that's not *100% wrong...
> I don't think tech debt should be communicated to the business people.
This is wrong. Technical debt is almost always incurred because of the business requirements. They need to be made aware the debt is being incurred at that time so they are aware it will need to be paid down ASAP.
Example: a startup is asked by an investor for a particular feature to be implemented for a trade show a week from now. That feature - if done properly - would require a database schema change otherwise the queries will be abysmally slow. Luckily, for the trade show the amount of data being queried is small enough that it won't matter. But as soon as it starts getting used, whoa boy.
Management needs to be made aware that there is technical debt being incurred here. Yes, the deadline will be met and the show will go well. But there better be time on the schedule to do it right afterwards.
>This is wrong. Technical debt is almost always incurred because of the business requirements. They need to be made aware the debt is being incurred at that time so they are aware it will need to be paid down ASAP.
I absolutely loathe doing this - the costs of technical debt are too specific (e.g. 2 hours a week) while the benefits are too diffuse and too hard to measure. If you have a conversation with business people about it it feels like you're having to make a business case for buying pencils.
It would be, but the most realistic, specifc claim you could typically make is that some hypothetical feature that probably hasn't even been dreamed up yet would get done in less time than it normally would.
It's a hard sell, and that's precisely why it doesn't get done.
Maybe I'm misunderstanding. I think "the benefits [of technical debt]" are the reasons for taking on technical debt. When you take on technical debt it's for a specific feature that's needed, not for something that hasn't been dreamed up yet.
In this case I'd tell the business that we can absolutely write a hacky version for the trade show, but if they want a fully performant version later, that's a separate story, and much of the work going in to the first story will have been wasted.
Maybe we don't even disagree on much there.
The scenario I had in mind was when you decide to rework/refactor some existing code because it's badly designed because of incompetent original design, changing requirements or whatever else.
To me that's just an ordinary Tuesday, all part of the work we do all the time, and nothing I need to communicate, any more than I need for when we take a lunch break.
Maybe this is the difference between thinking of the business as my "customer" rather than my "boss".
I'm not sure about that, most deadlines are arbitrary. There are exceptions but I wouldn't class a trade show as that. It doesn't come from business requirements, it comes from poor management.
I understand GP as saying that if business people know about technical debt, it means you're giving them too much details. Business people care about features being implemented and bugs being fixed; they don't (shouldn't) care if implementing a feature is 1/3 direct work on the feature, and 2/3 fixing the architecture to make the direct work on this and future features faster - but if you communicate this split to them, they'll start asking if you couldn't cut that 2/3 part out.
I mostly agree with this in theory, but there’s a tragedy of the commons problem when working on a development team. A developer who takes the quick-and-dirty route will appear more productive in the short term than one who tries to maintain long-term development velocity.
If they’re working on a shared codebase together, the latter developer will end up cleaning up the mess left by the former and getting less recognition from management. In all likelihood, this developer will either leave due to resentment, be laid off for underperformance, or stop doing the maintenance work. In any case, there’s no longer anyone seeing to the long-term development velocity and the team’s productivity will slowly grind to a halt.
I can think of a few ways of combatting this though.
1) Slack in the system, meaning there is time without explicitly defined tasks that have to be done.
Basecamp does something like this where they take 1-2 weeks off scheduled work after a 6-week cycle to let their employees fix things up [0].
2) A good definition-of-done plus a technical culture focused on quality.
My current workplace has a culture of valuing correctness and quality over velocity, it has been a nice change of pace from my previous workplace where velocity was valued over everything else, leading to lower velocity as our code became worse over time.
Yeah, I've been on a team like that, and I can only recommend either (1) leaving for a functioning team, or (2) fix your team. And I don't know how to do (2).
I'm talking about how the development team as a whole should relate to other parts of the business.
But business people know all about debt because it's common for a business to operate on debt. They can relate. And, as the advice usually goes, it's better to first address your debt before going into saving and investing.
The interesting kind of technical debt is when you realize you modeled the problem wrong, or the problem has changed and you need to change the model. A simple example is an auth system that assumes a one-to-one correspondence between account and user login. Then you get, for examples, agencies that are conceptually one account but have multiple logins (so each client can only see their own account). Oops, that breaks the model. Changing this requires database migration, code changes, UX changes, and so on. It's a conceptually simple change but has wide implications.
Business folks are more sympathetic to that kind of tech debt. The eye rolling is more around silly plumbing, version upgrades, etc, that add no business value. The React ecosystem is a good example to me. No two shops use the same stack. And alternatives like Angular are more "batteries included" and stable / "good enough to get the job done". But less popular.
I wouldn’t categorize this as technical debt unless you’ve deriberatly choosen to implement the wrong abstraction first, to get the product out the door earlier, or some other short term benefit. Technical debt is a deliberate action with some upfront knowledge of its consequences.
I get your point, but it feels like things have swung too far in the opposite direction. Compared to other engineering (chemical, civil, industrial, etc), we do a lot of change for little good reason. There's too many ways to solve the same problem, which kills "best practice".
Trust me, in 20-30 years, after the apocalypse, someone needs to turn on a water filter and npm can't download the correct package... and they die of dehydration... just write the Black Mirror episode around it!
> COBOL on a mainframe would remain current for decades
Only if the underlying problem solved by the program doesn't change for decades, and even if this case, the piece of software in question is a constrain to the industry using it. I'm sure a dentist would loves to have some fancy REST 4.0 BUZZ-WORD-OF-THE-DAY client management system.
I am using some industry "standard" industrial CNC software and it not only constrains me to 15 years old hardware but also restrict me to use programming quirk specific to this software which very much qualify as being "tech debt". In this case, the business is built around the software.
There's really a few distinct concepts here, and I think it's important to be precise.
- Keeping up with platform churn. How much of this you will do is really a CTO level choice, in terms of a) permitting internal platform teams to break backwards compatibility, and b) permitting adoption of trendy technology from outside.
- Knowingly trading off quality for speed to meet a deadline. This is tech debt in the strictest sense and you can quantify it as you accumulate it. Tech debt comes from a business / product management choice to force a project to release before it meets standards.
- Seeing better abstractions and decompositions, new test cases, observability hooks, etc. with the benefit of hindsight and experience. Examples: we found four bugs in this module this quarter and it's really complex, I see a simpler and less error-prone way to rewrite it. There's a lot of interest in adding new features with a certain shape and each time you have to touch code in three different places, let me consolidate those so you only have to touch one. People keep asking a particular question that requires painstaking correlation of different logs, let's add a new logging topic that answers it directly. Stuff like this. These are never strictly necessary and business impact is hard to quantify, but if you do this stuff never, you're going to have a lot of cruft in a few years.
I've got Javascript projects of a couple of years old. There are certainly things I'd do differently now if I were to set them up today, but I wouldn't consider that technical debt: those things are not giving me practical problems in the foreseeable future.
The only of such projects that I'm still running which does have significant technical debt, from my point of view, is the one I set up without a CI system and without a package manager with a lock file (those didn't exist back then) - I cannot update that code today, because I cannot build it. Rewriting that one with a reproducible build set up is something I consider tech debt and which is still on my TODO list of intending to solve one day.
Back then technology could set more boundaries than today. Most of these old mainframe systems I have worked with, use batch jobs (running at night when the system is offline) today being offline is often not acceptable and the users can't wait until the next job to have their data in the system.
But I agree that being conservative with pulling in new technologies and dependencies, new and good is not the same.
I think the biggest thing with tech dept is communication. We can do this in 1 sprint but then it will cost more in the future. If business always choose the quick matter how you put it and explain when it causes problems, yeah... :(
Disagree - "investing" is an activity that can be deferred without anything but opportunity cost; "paying off debt" expresses that not doing it has real consequences.
Well, paying off debt can often be deferred indefinitely too, as long as you keep paying the interest.
Instead of thinking of it as "debt payments" you could just think of it as high operating expenses for the development team, and invest in reducing them.
You don't even have to pay indefinitely, just as long as the code is still in use (for the subset of technical debt that has externally visible effects, such as performance issues) or even just while the code is still routinely being worked on (for the rest). At least it looks like that at first glance, which makes not paying so attractive.
But your alternative mental model is spot on, it's a costly hindrance to development, not a cost imposed by the code itself. And "only until you stop working on that code" is unfortunately just half of the truth, because a team that is used to living with a problematic style instead of dealing with it might easily end up repeating the patterns they abhor when given the chance to greenfield, because they don't know the alternative well enough.
I sort of agree with the author, for a similar reason that I don't like the concept of code rotting. Digital artifacts left alone don't change. They don't become more broken over time. Technical debt in code you're not currently modifying isn't incurring interest - and a debt without interest is one that you can safely leave unpaid indefinetly.
Both "code rot" and "technical debt" are concepts that, unlike their real-world analogs, are only problematic in context of ever changing environment. I feel this is a crucial and unmentioned difference that makes the analogy bad.
> Both "code rot" and "technical debt" are concepts that, unlike their real-world analogs, are only problematic in context of ever changing environment.
It should be taken as a given that code (and everything else) exists in an ever changing environment. Computers are physical machines; they run down over time. Capacitors need to be replaced, and you've got to blow the dust out of your fans once in a while.
Especially in web development, the environment changes quickly. Who could have foreseen that the authors of the JavaScript spec would introduce their own "remove" function? Not me: I left my code unchanged, and it broke. Or, more technically, it rotted.
Let's be honest - web development is crazy land. It's an outlier, by far the fastest moving area of our industry. And that's still only if you follow the "best practices"
Maybe I should've put it in a different way: one of the biggest benefits of putting things in a digital form is that the digital form is immune to decay, as long as someone maintains the infrastructure for copying and reading that digital data. The default state of digital data - including code - is not rotting. We do a lot of extra work to make it rot. So the question should be, why are we doing it, and shouldn't we embrace this feature of permanence that we get for free? It's not impossible [0][1].
Version pinning and only doing updates in explicit, planned steps, is a start. Runtime environment can be virtualized and thus made to outlive hardware (virtualization is a way to turn hardware into software, to make the former immune to decay). Web folks have Babel, which today is used to evolve the language faster than the browsers can keep up with, but it could be used for the reverse - instead of compiling ES1234 into ES5, it could also be made to compile ES5 into whatever backwards-incompatible ES-whatever the browsers will be running 5 years from now. This is how you restore the pristine state of code undecaying. The nice thing about the digital medium is that it only requires maintenance at the boundaries of change, and we get to move those boundaries - we could opt to keep them concentrated in few places (like virtualization software, transpilers, hardware abstraction layers), instead of spreading them out everywhere and then saying that the code "rots".
--
[0] - Common Lisp has many warts that make it suboptimal for modern software development, but one of the best things about it is that CL code doesn't rot much. When you pull in a CL library that's 15-20 years old, you expect it to run without changes. With 25+ years codebase, you might need some fixes, mostly because the language wasn't standardized until 1994.
[1] - Maintaining backwards compatibility matters too. I can take a Windows .exe compiled 15 years ago and I expect it to run on a modern Windows machine just fine. Arguably, this is one of the main reason for success of Windows - Microsoft put in work to ensure old software doesn't break in new Windows versions.
Rot and rust are unavoidable on the small scale without special effort. Code does not "rot" on a small scale; it only starts to matter once you start playing the IT industry fashion treadmill with the world at large.
Have to disagree with this. Unmaintained code is a massive security risk. Eventually someone will find a flaw in some part of your stack and your code is dead.
No, your code still isn't dead. It's revealed to be potentially exploitable by someone, somewhen, maybe. Code is not accruing danger or becoming dead by just sitting there, untouched.
Unless you've been particularly careful with your build chain, your ability to actually do anything with an untouched codebase degrades over time. There may well come a point where just getting the damn thing to build so you can release a version with whatever vulnerability patched can cost more than the remaining lifetime value of leaving it running. In that case, it's already dead, it just doesn't know it yet.
What you say "being particularly careful with your build chain" was the status quo just few years ago, and I'd argue is still good engineering. You should pin your dependencies, and updates of everything - including build tools - should be explicit steps in the process.
This should be trivial on the small (just pin versions and disable automatic updates of your build tools / package repos), and with recent proliferation of cheap VMs and containers, it should be simple over time - you should be able to pull a 5 years old image full of old software and rebuild the original binary.
> There may well come a point where just getting the damn thing to build so you can release a version with whatever vulnerability patched can cost more than the remaining lifetime value of leaving it running. In that case, it's already dead, it just doesn't know it yet.
Vulnerabilities aren't sins, they are things that can be fine to leave unfixed. The software isn't dead, it'll just keep running with a known exploitable issue.
> You should pin your dependencies, and updates of everything - including build tools - should be explicit steps in the process.
Yes, and in the not uncommon case where part of the build environment is left unspecified, the untouched code accrues debt over time. As you say, it's possible not to take on that debt by making a significant investment up front. This isn't an argument against the debt metaphor, it's an argument for it.
> Vulnerabilities aren't sins, they are things that can be fine to leave unfixed.
My point doesn't change if you s/vulnerability/sufficiently serious vulnerability/.
It does, though. A single line might not change - but the world around it is always shifting. Libraries in frameworks, Operating Systems, drivers, external data structures - nothing stays static in the environment our code lives in. As long as code is running, it needs to be maintained. (Most of the time it can be negligible, sometimes serious...)
Most of the things you mentioned are a choice, not something inherent to the code as an artifact. If the external systems your code interfaces with change, you need to change your code. Other than that, updating your OS, libraries, drivers - these are all choices. You can not update them, and the code should run exactly as fine as it did when originally written. You can virtualize the environment to mitigate the risk of the underlying hardware platform going down or you. Code decay, to me, is mostly self-inflicted.
It's a feature of the dependency system and speed of change of that ecosystem. A large number of systems have been written with these 'high maintenance' stacks in recent years.
I agree the users of these stacks are often short sighted and you could call that bad engineering. However it's very much a feature of those platforms, not so much individuals decisions on then.
Personally I prefer ecosystems with large common libraries that are highly stable and cohesive to avoid those problems .. but the reality is so much software is built in fragile/high maintenance stacks these days, npm and react being the worst bit not only examples of extreme code rot from time decay of the ecosystem. Try updating a react app from two years ago to really understand..
That's how I think about it, and that's hopefully how a lot of people think about it, but I suspect for most people that's not true. For a lot of people debt isn't nearly as scary as it ought to be, and the idea of, instead of paying off debt, spending money to theoretically make more money seems like much more of a sure thing than it actually is.
Perfect example: I'm in SV, and I bet there's a not insignificant slice of the population around me who think that the cure to budget deficits (i.e., debt) is to reduce income by cutting taxes, based on a mystical belief never seen in reality that the growth fueled by the tax cuts will result in both lower taxes for the population and higher tax revenues for the government (i.e., investment).
I'm almost 50 years old, and there has never been one iota of truth to this notion in my lifetime. Not one teeny tiny iota. But some people just can't resist going back to that well over and over again with the same results every time. The notion that you can kill two birds with one stone, and both ignore debt and create wealth at the same time via magic Austrian fairy dust must be very powerful. It's like a black hole of self-evident idiocy that people keep flying into for some reason. It must be a form of cognitive bias, but if so I don't know what to call it.
I have for the most part stopped using the term technical debt, especially when talking to non engineers. There are many reasons why I believe it is an exceptionally poor term, not the least of which being that, unlike monetary debt, technical debt is not quantifiable. Imagine approaching your CFO and convincing him to take out financing to fund an upcoming project, and bring completely unable to tell him the principal amount, the interest or the terms of repayment. I’ve certainly never seen any organization buy into the metaphor deeply enough that they were capable of reasoning about it with anything close to such financial permission.
The term I prefer and use quite frequently these days is “technical health.” As in, choosing to continue to the next project without refactoring the new features that we rushed to deliver will negative impact technical health. Technically unhealthy code or services are slow, complex, unreliable, not performant and cumbersome. Technically healthy services allow us to be nimble, to make changes with ease and speed. They are easy to work with and easy to reason about, particularly when things go wrong.
It should be quantifiable to within an order of magnitude or so, although possibly not in advance. In my experience the problem is that any individual piece of technical debt is too small for it to seem to be worth the effort for the difference knowing the value would make.
When anything except new features is called "tech debt" I think Ward Cunningham's analogy has been stretched too far. It's supposed to mean forgoing needed work now with the intent to do it later. People should balk at unexpected bugs or unplanned refactoring being called that.
Occasionally the new feature's debt is never paid. I'm cleaning up a location which has had this happen 30+ times over 20 years. It resulted in 15+ sightly different ways of doing the same thing. Much of this without a function call in sight. I've been splitting functions out of it slowly reducing the 80+ variables into discrete behaviors. It's now unit testable.
I'm often asked why can't I add something quickly, pointing to this file is more than enough reason for all but the most important projects to agree to wait.
It is tech debt, it just wasn't taken on as a debt intentionally. It's like going wild with a credit card - it's debt, but there was no rational thought as to what the consequences would be or how it would be repaid.
They're both good. Stop trying to reduce a multivariate problem to a single variable problem. Managing a software project, leading a team, running a company, raising children are multivariate problems. There is no right answer or best tradeoff in all circumstances. Everything is highly context dependent and there's a balance. You have to walk that line and be careful not to stray too far to either side in everything. You can be too assertive, you can not be assertive enough. You can focus too much on technical debt, but you can also not spend enough resources addressing it and have it cost you more in the long run. Find the balance. Thats what leadership, and more broadly, life, is all about.
IMHO there are parallels with "physical" engineering if you look carefully. Here are some examples:
1) We figure out that asbestos is bad, mmkay. Now suddenly half of the buildings have a need to perform expensive maintenance work even though their business needs for "features" have not changed - that's analogous to finding a security vulnerability in a subsystem or protocol that's not even your own but that your system uses, where fixing that may take a lot of work.
2) For electric wiring. You may have a building wired properly according to code requirements as it was in 1950. Nowadays we have a better understanding of what's safe and not, and while you're often not required to fix that immediately, in many places you wouldn't be allowed to "just" do minor changes - if you do alterations, then after you're done that place must be meet the current code requirements, which may require extensive expensive work. This has a clear analogy to the type of technical debt caused by relying on an obsolete version of some dependency - adding small new stuff may force you to do integration work so that your system works with a newer version of that dependency.
3) Another example of electric wiring - you may sometimes encounter improper work done that way to save time or money because of some urgent deadline. And you can't just come in and make up the missing parts later - fixing it often requires simply ripping all of it out and putting in the wires properly; so if it's not a temporary place that's up for demolishing soon (just as with software, temporary structures also sometimes stay used much, much longer than planned) then you need to re-do it eventually if you want to keep using it safely. And there are cases just like that in software.
People handling construction and reconstruction projects, as well as all their customers, have some understanding of these concepts when it applies to the physical world, and it's not that surprising to them (well, the costs for the tasks mentioned above are sometimes unpleasantly surprising) - so perhaps we should frame it in similar terminology to piggyback off of these analogies instead of debt?
Or the company can't source the same materials / quality as before (supplier change, company moves, etc) and need to go into trouble fulfilling existing orders with the old design (paying extra both time and money for the old material) while trying to sort out a new design with the new material available...
Yes, companies sometimes have to redesign their product (costing a bunch of time and money and potentially making it worse) because they can't supply some ingredient or technical part anymore and have to use something else... which is pretty much the same thing as having to rework your system because a software dependency you're using isn't supported anymore.
Furthermore, you may simply be working with a supplier where you know that they're stopping production or going out of business soon, and you know that you'll need to rework your physical product soon but haven't started doing that yet, such cases happen, they'd show up on internal risk assessments and shareholder reports - and I'm not certain how this class of challenges as a whole would be called in the (non-software) business world, but it's definitely not "debt".
Debt:
* Interest is due on a periodic basis, with consequences if you default such as recovery, court, reposession, bankruptcy.
* You may be asked to pay back the principal on a trigger (e.g. margin call) depending on the contract, or you may have to pay towards it each month (repayment mortgage)
"Tech Debt":
* No pressure to repay
* There is arguably a recurring cost to having bad structure, but this cost varies based on the amount of activity on the codebase. No new features and only security bug fixes = very little interest and no new tech debt. This is unlike regular debt.
* Can't really be measured accurately (like real debt).
* If the company is sold it is not on the books.
In my experience, technical debt is much more about time-to-market minimization than it is about reducing costs. Here in Silicon Valley, raising capital is usually not the limiting factor. However, the spoils to be had by being first to Market are so great that they justify pushing a lot of pain into the future just to be the first to get the product out the door and get the first-mover advantage.
I've observed that often programmers in the trenches I don't really understand the dynamics of the business well enough to be trusted to make critical decisions regarding these trade-offs.
In a way, one could argue that there is no such thing as "technical debt".
This term usually denotes technical decisions which lead to functional software at a cost of now difficulty in future development, debugging, performance etc. But that cost only matters only when it's actually collected, e.g. when later development is actually difficult because of those decisions. Which might never come to pass, and activities to"improve" the codebase are wasted unless one is 100% certain that it will happen.
It might be better to consider that practice "fixing non-functional bugs".
I got hired as a project manager to oversee the 5 programmers on a massive project once. Their previous “supervisor” was the owner/ceo with NO technical background for about 14 months.
Every single feature and block of code was written in 30% the time it should have been given. The ceo had no idea the technical debt he was creating until it was too late.
The day I came aboard any new feature or change that could be completed in 100% standard time needed 200%+ the standard time to account for the shortcuts taken over the previous 14 months.
How did you approach your role as PM, aware of the tech debt but also reporting to an unaware CEO? Power through with new features at half the speed, or try to convince the CEO to allow refactoring/maintenance?
Sorry for the delayed response (no notification on mobile).
For some reason my relationship with the ceo was viewed as trusting whereas their programming team was not. I was able to get the time required (in 9 out of 10 situations) by explaining in detail why it was required.
We actually had a complete rewrite in the pipeline - an architecture written by myself - that was on the table. But they shut down before that was possible.
I don’t regret taking the position despite the frustration and difficult inherent to what I had to tackle. I got a very closeup view on technical debt that has assisted me throughout my entire career.
Our marketing team understands the power of framing a concept in a favorable light as they are constantly correcting our use of "eating our own dog food" to "drinking our own champagne". Of course, I paraphrase this to "drinking our own dog food".
I agree. It’s amazing how important a choice of words can be. The word debt has negative connotations for many people, perhaps even hints of irresponsibility (e.g. you should have done it properly in the first place). It’s not always possible or efficient to educate non-technical colleagues, so I find that using more business oriented language really helps with getting that work accepted as meaningful. Even within the team, highlighting the benefits of the work leads to a more positive approach (although at the developer level we love removing terrible code, so don’t need much motivation).
The problem I have with opportunistic refactoring, in the context of Scrum at least, is usually the extent of the cleanup is not known until you start making changes. By that time, you have already given an estimate for the story points. Hopefully, the business considers this a forecast. Unfortunately, many times this is considered a commitment. If you do the refactor, you may miss your commitment, and easily measurable things for the business (your velocity) look bad while something that was not easily measured (tech debt) remains unquantifiable and unchanged either way.
That version of Scrum is horrible, and should be fixed. Scrum switched "commitment" to "forecast" in 2011, and it should be well known by now that you can't compare velocity across teams, or even within the same team out past a couple of sprints.
Define "Investing in improving structure" - Of code? of system architecture? of teams and of company?
Define "is x better than y" - are we talking about comparing two different activities, or is the question about different ways to name the same subject?
Is it better because it's a more productive activity, or because "if you do y, you'll also get x as a side-effect" ?
I'm sorry, without much clearer phrasing, the question is hopelessly ill-posed to me. Can't answer something that could mean anything.
I don’t believe you can answer that question... or that this question has an answer. I read a few assumptions that technical debt can be due to bad coding practice. While it could be, in my experience it often comes from business goals changing, rendering some past infrastructure obsolete or improper. There is also a lot of value for engineers to deal with debt. If dealing with debt is hard, it probably will make your team better.
I only use the term "Tech Debt" among engineering team. For communicating to a broader audience, I use:
"Improve product, engineering, and operational excellence".
The idea is that the broader audience focus is on the overall benefits and costs, not on whether we have debt. This phrase also allows me to make process changes that improves the team's efficiency outside of Tech Debt.
That We have the ability to have technical credit to draw down on shows the power of software and is valuable because it means you don’t necessarily need to build robust code to get something up and functional and it gives you an opportunity to learn.
Recently I was asked to come up with a word for this kind of thing and after some thinking I settled on 'reliability'. So now we sometimes spend a bit of time on 'reliability' which sounds pretty nice, doesn't it?
Sometimes, and it also drives better decisions that way.
If you have code that’s janky in some way but requires little maintenance or will be outright replaced in the near future, letting it remain janky is a reasonable decision.
Lately I've been talking about technical rent, for instance updating a local fork of an external codebase, and feel like that distinction helps my group more thoughtfully use the technical debt metaphor.
In financial terms, it’s more like you sold an unhedged put option.
I.e. you may be required to pay the difference, at some point in the future, not of your choosing.
The problem here is the mischaracterization of what technical debt is.
Most of the time I hear programmers talk about technical debt, what they really mean is profiling and performance (because over time requirements change and code size increases) or code "cleanup" which is usually a catch-all for "not how I would have done it".
Performance issues are real, but those should just be scheduled and tackled as needed. Code cleanup is quite often a red herring.
But technical debt is entirely different. It's exactly what the name implies: debt. We would have done X to implement a given feature or customer request, because there are many conditions and other systems to take into account. But there wasn't time to do it that way before the required deadline, so we did something else instead that would get by for now, but we all know won't last.
That is technical debt.
Sadly, often times management just see the issue as done on the schedule and there should therefore be no need to go back and "do it again".
Always pay down technical debt when you can. It will be paid eventually, it's just a question of how painful it will be when you do.
I think that may companies need a fundamental shift in managerial culture before they take technical debt even remotely seriously.
The only way I've ever managed to do anything about technical debt was by hiding the fact that I was doing it - since it was completely impossible to get any level of managerial buy in or even permission.
It shouldn't have had to be that way, I should've had higher level support, other developers to work with and competent testers - working in that situation ultimately left me never wanting to write software for anyone else ever again.
It's a disease of reckless incompetence, and it permeates the managerial structures of large swathes of the software development industry.
Same here, I'd always pay off a little debt here and there and the busy-waiters would be circling my desk asking for status -- from the busy body product owners to the nitpicking directors at their weekly inquisitions, there's a whole class of parasites whose only contribution is to trumpet how late we are and insist on asking 'how long will this take?'
I got it too the point where the standard changes took a fraction of the time to implement compared to what it would've been operating on the initial code base.
In fact, some of the functionality that I ultimately implemented would not simply have taken longer, it would have been impossible to implement without introducing world ending bugs.
None of that mattered. Nothing that I ever said or did in that place meant a fucking thing to anyone.
Where have you been all my life bro! Nobody gets it any more -- precisely the attitude needed in this industry; we need to turn the tables and take our craft back in-house
How's that meaningfully different than performance and code cleanup cases? Frequently, a team know how to implement a feature in an efficient manner, but only has time for a low-performance hack job. Similarly, "code cleanup" is as much a code word for "not how I would have done it" as it is for "bad architecture that will start constraining us very soon".
Performance work is no different than new features: requirements change.
Code cleanup can mean many things. When talking about architecture, it could be a legit argument that due to requirement changes there needs to be a rethink. But that's not debt. That wasn't a conscious decision made in the past to do A instead of B knowing that B was the right choice, but not realistic at the moment.
When talking about large code bases that have been around, code cleanup is a young person's term that tends to imply they are smarter or know more than the previous set of programmers who worked on it. That's possible, but a bad argument. And it's insulting. Obviously the code has worked for quite a while successfully, so it can't be as bad as it's being made it to be (that's management playing devil's advocate). If it hasn't been working, the schedule should reflect fixing it. Maybe that requires a massive recoding, but maybe not.
Cleanup is not about how the code works at runtime, but how well it supports change. An abstraction that looked right at the time shows its cracks years later. Changes that should be simple and low-risk become intense and scary when they need to fit in with it.
It's not insulting. For one thing, the person proposing the change may very well be the original author. For another, you have the benefit of hindsight and access to information they didn't about what bugs would emerge, which features would be demanded, and how hard it would be to implement them. Of course you're going to see better ways of doing things than were visible on day 0.
Increasing speed and reducing risk for all future iteration is some of the highest impact work you can do; rather than shipping a single feature, you're a force multiplier for all the features developed by your team.
The only time it makes sense to skip cleanup is when you're content to just let the thing run, and don't see any changes in the pipeline.
> That wasn't a conscious decision made in the past to do A instead of B knowing that B was the right choice, but not realistic at the moment.
I'm thinking of cases where it was a conscious choice. $deity knows that I've made plenty of such decisions in my career - chosing a hacky structure A to meet a next month's release, while articulating to management that after that release, I'll need to redo it to structure B. Sometimes a bunch of high-priority requests took precedence, which made the system stick to A for far longer than it should. I understand that to qualify as both "technical debt" and "architectural issue".
I use code cleanup as a term very differently, as a programmer with 20 years commercially developing software..
My process often involves creating layers/versions of the code that improve till the stage it's production ready. It might be half a dozen or more iterations to get to production quality for complex code, and the latter versions involve a lot of 'code cleanup'!
This includes things like implementing TODOS, adding logging, and refactoring and refining the concepts, the naming, making the code more dry, better srp and generally elegant code I hope that someone else would be happy to inherit and maintain and extend. That someone is often me!
It's taking something that works, mostly, but has limited clarity and elegance and higher future costs to work with, and 'cleaning it up'. Like a home or worksite cleaning up after working gets you ready to use it or work there productively in future.
I do this in most codebases I work in for awhile as well, when there's clearly need for improvements in dry, srp, conceptual patterns and clarity.
Our industry jargon calls this refactoring, but cleaning up actually relates to other stakeholders than the highly trained engineers, so I will generally call it cleaning up.
I agree with you that newer engineers often want to rewrite existing working things for many wrong reasons - but I think the specific term code cleanup has a place.
I think it's interesting that you've noted it as a code smell for a junior about to wreck a significant amount of company productivity. That's an all too common problem these days as the majority of engineers are junior with less than 5 to 10 years due to larger numbers entering the industry. I'm interested in the 'successful' strategies experienced engineers have used to head this off at the pass in organisations with non technical management who often listen to the loudest voices if they can't truly understand the issues themselves.
>> "That's possible, but a bad argument. And it's insulting."
Firstly, there are people smarter than me, it's not insulting, especially if a person makes a real improvement. Or maybe I made a mistake.
Secondly, original designer had to make a decision with limited information and time. Now we have the benefit of hindsight, so coming up with a better one is hardly surprising.
It can be in certain cases, but it sometimes depends on the perspective. Abstractions are hard and usually need enough time to think about and several iterations to get right. They are usually much more obvious in hindsight.
Because "performance" is not related to "functionality". If you skip corners (eg. panic instead properly forwarding an error, hack a new implementation over an existing interface, etc), you induce a long term cost each time you have to figure out why the code is misbehaving.
Technical debt is all these architectural issues which incurs a long term cost because minimum deliverables had to be met. Also, the first response to "performance" is always gonna be "throw more CPU/memory", not "let spend a month dev-time to rearchitecture." Tech-debt is only getting addressed when hardware can't meet the requirement, "performance" is just a marketing argument to sell it to management.
Making code perform more efficient is easily analyzed as a feature. If the inefficiency results in end user delays then it can be directly traded-off against other feature requests, if it only impacts costs it can be fairly easily be determined whether or not it is worth devoting (very expensive) developer resources against.
Prioritizing improving performance for aesthetic reasons, that is because a "low-performance hack job" offends someone's sensibilities, likely means either: 1) the team is poorly lead or 2) doesn't have enough high value work to do.
Debt can also be taken on accidentally. You can build a couple of features one way and expect future features to be similar, but then reality bites and the way you've built them makes the next feature that happens to come along harder, and the next, and the next. Or the way you've built them doesn't allow for the rate of change in that area - something in code wants to be in config, or something in config wants to be in a CMS. In the debt analogy, it's like having a variable interest rate: you can make a decision which is correct for today's conditions, but the market moves and you're paying twice the interest and suddenly it doesn't make sense any more.
I’m dubious of defining technical debt merely as the result of previous decisions taken to trade off delivery time for maintainability. That’s the responsible form of technical debt and responsible people will incur it. But there are irresponsible forms of technical debt just as there are irresponsible forms of financial debt. What makes something technical debt is the “interest” in terms of maintainability that one pays for it.
To make the term more useful to me I like to extend the analogy and think in terms of interest paid on technical debt.
So you took a shortcut because of pressure to ship but now you’re spending loads more time providing tech support. That’s the interest you’re paying on the technical debt.
So in terms of choosing what technical debt to pay off first consider which ones have the highest interest payments. And remember that like financial debt, technical debt also sometimes comes with introductory interest free periods.
Yes, it's extremely strange that people are looking for new terminology because they've been using the existing, perfectly good, terminology incorrectly.
Your third example is just some combination of the first two.
Performance issues: the efficient solution is too complex, we don't have time.
Code cleanup: this is not the way I would've done it if I had more time.
The debt metaphor fits so well because you're borrowing time.
Do people in other professions have the same issues? If the salespeople say this thing must be done in this particular way, or if the lawyers say this thing must be done in this particular way, are they questioned and overridden by their managers? I honestly don't know; it feels like software engineers are given a special level of distrust but it just might be that I'm perceiving that because I haven't experienced the other jobs.
FWIW where I currently work, we have an amazing middle-manager / product owner who has to fight the same battles with upper management with regards to process improvement and technical debt. She gets it, but has to fight the same battles we do. Sales and C-level want everything now with as little investment as possible.... I wonder if I just answered my own question?