never ask for permission for things you know are vital to your work
I agree to an extent. It can be easy to fool yourself about what constitutes good code -- in the sense of making a product better or work on it easier. Sometimes bad code is better left as is. Even working totally unconstrained, I prefer not to refactor something unless I have a pressing reason in mind.
My rule of thumb is this: as a programmer and an employee, I am professionally bound to produce quality software efficiently. If I know I can complete an assignment faster (or in equal time, but leaving behind a better code base) by rewriting something, building a tool, fixing something architectural . . . I will silently do it. No point in asking permission. It's in my charter.
On the other hand, if I want to take a lot of time to rearchitect something -- an order of magnitude more than it would take to just do whatever it was that brought me there -- at that point, it's a strategic decision and management deserves to know about it.
The way I see it, management has no right to require me to produce an unprofessional product in my day to day work. And I have no right to force management to use engineering considerations only in strategic decisions.
Then again, refactoring without permission could be a really bad idea. You need to ask yourself a few questions before you proceed.
Do you have thorough unit tests for the code that you are trying to refactor? If not, be aware that there is no way to know for sure that your refactoring won't break the functionality of the code.
Suppose it breaks the code. Have you thought about the operational impact to clients and the financial costs?
Let's say the costs are low. How big and political is your organization? What kind of trouble will you find yourself in? As the hysteria rises, will you be fed to the dogs over this?
How bureaucratic is your company and how many people do you need to interact with to fix a functionality breakage? The more people you will need to interact with, the more damage you will do to yourself and your reputation. Others will resent working in panic mode to clean up after you (now widely known as the "rogue" programmer).
Not to mention if there are other people modifying the same area of code in a parallel project. They won't have tender words for you when they try to merge back their changes and find it has completely changed.
Cleaning up code is a good idea but you need to be aware of the context. Always do the best you can.
+1. My most common conundrum these days is whether to favor improved quality or consistency. If I make improvements to the current project, how can I make sure they make their way into other, parallel projects that share code and architecture. Is it better to be "better" or "worse but consistent?"
Any refactoring party on some real messy code should consist of a long chain of little proven-safe logical steps like the one above, or pure renaming, or move-around. In the end, like when the sea is gone, you may find some smelly dead animals that you'll have to take bare hands and fix, or not (sometime it is simpy too heavy to move, you'll have to leave it there, under a thick isolation layer).
Very fair considerations. I think it is easy, as a programmer, to think of refactoring only in terms of cost, and to ignore risk. That is a mistake.
My own rules to keep myself honest in that regard are these:
- Never rewrite anything you don't completely understand.
- Never rewrite anything beyond what you would have had to touch to get the job done anyway.
- Never rewrite anything just because it bugs you. Have a compelling, practical, immediate reason why fixing it will save you time right now.
And my way of work certainly reflects my environment. I do a lot of work on code for which there aren't test cases, aren't requirements, aren't other developers looking at it, and isn't even a very high probability that it worked as intended the first time. In some cases, I'm even the sole arbiter of what "works right" means.
Certainly, the higher the quality of the code, and the more other people--developers or otherwise--depend on it, the more lightly one should tread.
Do you have thorough unit tests for the code that you are trying to refactor? If not, be aware that there is no way to know for sure that your refactoring won't break the functionality of the code.
If there are no unit tests ... the original coder didn't care about correctness, and refactoring towards testability is a good thing.
The more people you will need to interact with, the more damage you will do to yourself and your reputation. Others will resent working in panic mode to clean up after you (now widely known as the "rogue" programmer).
I'm in two minds about this. Yes, you will need to be careful about not introducing changes, and yes, occasionally fixes will be needed. But better code should be a plus to reputation. If it's not a net plus, be more cautious. If that can't be done, I'd be thinking that I was in the wrong job, and my reputation with those co-workers was irrelevant.
I totally agree. I recently wasted a whole month refactoring some code (not without permission) whilst working on a defect so that the 4 main UI's of our product had roughly the same code structure -- even if that code isn't actually shared between them it makes things infinitely easier if they behave similarly. Unfortunately, although the refectoring fixed the defect and allowed us to progress with some features, it broke lots of other vital functions and after 3weeks I had to make the call to revert back to the original.
My new plan to fix this massive code smell is to leave the company :(
I'm facing a situation where there's a part of some code that I'd like to improve: and there are design flaws in the old code that will make it more than just a simple refactoring.
Although the subsystem is a small part of the system, it's used in many different places, and it's complex enough that I know that functionality will be broken. It's the kind of system that would really benefit from TDD but it's not practical to cover all the code that would be affected. There would be no substitute for a thorough round of acceptance testing and whack-a-mole trolling for bugs.
So my manager and I have to make a call about the benefits and risks of this work, which in turn connects to project management issues. I wouldn't want to do this late in the schedule, but would feel more comfortable doing it early. At certain times I might be able to do this work and have very little impact on other developers but at other times it could impair their ability to do work. (Yes, intelligent use of version control could help a lot here, but that takes coordination with the team.)
The point is that it's a team effort. There are some cleanups which are easy and low risk and you should go right ahead and do. Yet, all programmers have a way of underestimating just how much a twisty piece of code that they don't understand does, so you've got to work with your manager and the rest of your team to control risk.
I think it is worth making the distinction between refactoring and rewriting. I tend to take the strict view that refactoring should not change behaviour, so for example if a unit test fails, and you refactor some code, the unit test should still fail because the code should still be doing the same thing.
Since you say that you were fixing a bug, and the length of time it took, my guess is that what you were doing was a rewrite, not a refactor.
Someone else had a good line about refactoring, though I don't remember who it was: if you're refactoring code, and it isn't unit tested, you are not refactoring. You're just changing shit.
I hear you but all this means is your refactoring wasn't good enough - maybe you underestimated the scale of the undertaking. I have been there.
Don't lose faith - just document all the features, ideally with unit tests, before you make the next fix attempt. Leaving the company is of course a solution too, I'd just offer them to take even more time out to really fix that code. I do agree that living with the bad code is not acceptable though.
Only if you're extremely confident adequate testing is in place – in any form, be it unit, smoke, user or whatever – to catch any potential regressions. Using version control does you no good if you don't catch the regression and roll back before customers are affected.
Oh heck yeah. We at HN are immersed in tech. Our experience is with companies whose core domain is tech and take things like source control for granted. But what about your local gym or sports franchise or restaurant chain or newspaper? There are millions of companies that don't have any kind of technical director or direction. Where some manager thought it would be a good idea to have a web site, and hired a programmer who's measured on how fast he can throw it together. What's a testing server? Heck, what's a server?
The other thing that needs to be considered is the total effort for the refactoring including necessary testing.
As a development manager the thing that scares me isn't the small fix, it's that it takes place on a critical much used library that has potentially far reaching consequences.
I had experience with multiple such examples. At the same time I was developing new functionality, my boss was also tweaking the libraries. So our tester kept telling me my code was broken. Then I'd look into it and find that the Strategy object used to calculate quantities in the whole app was broken. Again
Have you ever heard of unit tests? It's the only solution.
Unit tests, of course, are another hard sell. "what you need to spend just as much time writing tests as you did writing the code??"
But I find the benefits are almost immediate, at least for new code. It's not some abstract thing down the road - it's an immediate boost to dev speed.
I agree that producing clean code is part of our charter, but sometimes we're put in situations with close deadlines and a boss with no understanding of the importance of clean, organized and sustainable code. For example, I had a boss which would regularly check on my progress and wanted to see what I was working on and he'd say "that looks good, let's wrap it up by today", my typical response would be "it's not done, still a lot of rewriting a clean up to do" to which he'd respond "does it work? I don't want to spend much more time on this". I would then try to work really fast to clean up my code and finish up the functionality. I was forced to push through stuff that wasn't on par with my standards.
I learned how to deal with this by hiding my process. I wouldn't show all of what I had. This made things easier. I still met deadlines and I felt much better about the code.
Agree completely. Also the same point of other sources:
From 37 Signals' Getting Real[1]: You need people who are passionate about what they do. People who care about their craft — and actually think of it as a craft
And from Software Craftmanship Manifesto[2] :Not only working software,
but also well-crafted software
Not only responding to change,
but also steadily adding value
Not only individuals and interactions,
but also a community of professionals
Not only customer collaboration,
but also productive partnerships
Excellent points to add to this discussion. One must always weigh the time it would take clean things up.
I think the main point I was trying to make with my post was; a lot of times it would take less time to just fix things than it would be to ask for permission to do so.
The corrollary of this is that it almost always takes longer to explain to a Project Manager why something isn't done yet/can't be done/shouldn't be done that way (etc) than it would do just to finish it or do it another way or do it properly (as the case may be).
I mostly agree, however I think it is important to highlight that these considerations have a real-world impact.
One of the biggest fallacies out there is that good software engineering is about aesthetics and somehow detached from the resultant product, e.g. 'the user doesn't care about how it's implemented' etc. - this is patently false, as code quality dictates robustness, maintainability and ability to make changes to the code base, both during development and after shipping the thing.
This is something that took me 2 years or so to learn. One day I realised nobody was really looking at my timecards in depth so I started allocating extra time to things and using the extra time to fix the things I thought needed fixing. Once I started delivering on this I showed my manager who agreed that it was a good use of time. I was given free reign to fix anything I felt would add maximum value, provided the bug fixes continued to be delivered without any major compromise.
Since that time I have refactored quite a few of our codebases; added unit tests, fixed some build processes, improved performance and generally feel happier at work for getting things done that are important to me.
Dont get stuck in constant bug fix mode would be my suggestion. If you cant get approval to fix things then change jobs because bug fix after bug fix is depressing and will bring you down.
Yep - I learned it in my first job after college. We had a hacked-together app that was 99% complete. It then remained 99% complete for the next 2 years as we fixed bug after bug after new-bug-that-was-introduced-by-the-last-bugfix.
The codebase was unstable from the very beginning. It did eventually ship when we had added enough duct tape, but it never turned into a solid app.
Towards the end of year 1 I already knew that most of this simply needed to be re-written and done differently, but I was overruled by more senior folks who were responsible for this mess. It was always "we can't do this now, because we need to ship as fast as possible".
Sure, but considering this is the author of the article we're commenting on, I think an exception is reasonable here. I'm interested in which comments the author's aligned with. It's a discussion, after all.. one that this person started. I'm not saying vote up.. but we could lay off the down arrow.
The reasoning behind not cleaning up your code is that if you wait long enough, the problem will often go away - literally! Technology moves fast enough that your feature is usually obsolete within a year or so; if you're not on the critical path that everyone is building upon, chances are your project will just be canceled and all the time spent polishing your code will be wasted. Better to get stuff out there so you have a better chance of being on that critical path, and dealing with the inevitable messes and complaints of "this is shit code!" later.
This is the "cascade of ADHD teenagers" development methodology, which is much maligned by professional programmers, but actually seems to work quite well. Google, FaceBook, and Twitter all use it to varying extents, and the entire valley startup ecosystem is based around it.
> Technology moves fast enough that your feature is
> usually obsolete within a year or so
That depends on your industry. My company has some core infrastructure running on code written in the 90's when characters were scarce (8-character variable names) and actual loops were even scarcer (goto's instead).
Feature-sets like "processes transactions" never go away just because technology moves forward. Some of that almost feels like you've had "too much of the Kool-aid."
My company barely existed in the 90s - it was some research project at Stanford.
And feature sets like "processes transactions" absolutely do go away because technology (and its applications) moves forwards. This was a large part of the success of MySQL: relational databases started getting used in applications where transactional integrity really wasn't that important, and they were better off working around it (or just showing inconsistent data to the user) rather than paying the price for transactions.
Or look at this recent discussion surrounding Google Megastore:
Megastore suffers a large latency penalty for ACID semantics. In many applications, it's better not to take that latency hit and instead settle for eventual consistency instead.
Financial apps are a different story. I interned on Pay.gov in college, and then spent the first 2 years of my career at a financial software startup. The standards are much different when you're handling money (or avionics, or medical devices, or anything that's "mission critical"). And even in the finance world, the standards vary a lot depending on whether you're doing analytics or transaction processing.
But while that software will never go away, it becomes a smaller part of the whole software industry. Much of the interesting growth is in areas of software that have nothing to do with money, other than being able to make it. It's silly to apply the same standards to software that processes $10M/sec (as one program I once wrote did) as to software that lets you throw virtual sheep at each other.
Arguably, this is just that new domains appear or grow from what used to be niche areas into big ones. The old problem domains requiring ACID transactions still remain.
If you look at some of the big investment banks and hedge funds, they won't bother refactoring; they'll just throwaway and re-write.
Their architecture is designed to be really modular and very loosely coupled. So you there is little value lost in throwing away and re-writing (ie. You won't throw stuff away thats working fine with the studd you want to re-write).
Technology moves fast enough that your feature is usually obsolete within a year or so; if you're not on the critical path that everyone is building upon, chances are your project will just be canceled and all the time spent polishing your code will be wasted. Better to get stuff out there so you have a better chance of being on that critical path, and dealing with the inevitable messes and complaints of "this is shit code!" later.
I cannot disagree more strongly. I'm dealing with a decade old application that's been coded using this philosophy, and there are enough code smells here to make a sewage treatment plant seem fragrant.
There are methods many hundreds of lines long, inside classes that are thousands of lines. There are isNegative() functions that return true for values above ten million. There are hidden dependencies all over the place, as people patched in functionality in the wrong place and no one moved it to the right place later. And, of course, there are no unit tests.
The impact of all this is that even trivial changes take weeks, as the developer has to wander a morass of bloated functions and tangled dependencies in order to just find the source of the issue. The "cascade of ADHD teenagers" can work for a couple months, but its not a long term strategy.
To note, none of your examples use that methodology any more. I'm not sure if Google ever used it. Facebook and Twitter may have been like that at the very beginning, but both exhibit quite disciplined coding standards today.
Google totally uses it now - sometimes. I've been specifically told, by a VP (who was one of the early employees), not to refactor code which very much needed it, because the engineering resources were needed elsewhere. There've been other times where I've made the call on my own not to cleanup dirty code, and it was 100% the right decision, because we never touched that code again. There've been still other times that I look at code Jeff Dean or Craig Silverstein wrote and say "This is a piece of shit" - encapsulation violations all over the place, an API that encourages you to do the slow thing (after micro-optimizing for efficiency everywhere else), complicated object ownership semantics. And then I remember that they probably wrote it in a day, for requirements very different than we're facing now, and cut them some slack.
The operative word there is "sometimes". It's a very useful skill to be able to make a judgment call and say "This code is peripheral; we're unlikely to touch it again" or "This code is temporary; we'll replace it in 3 months" or "This code is critical; it better be rock solid, because 500 developers will be changing it every day and learning how to program from it."
The point the article makes is that a manager doesn't have all the information, because they're not the ones who look at the code all day. And the point I'm making is that a developer doesn't have all the information either, because they're often not aware of the opportunity cost of fixing that code. What other features could you have implemented in the time it takes to write those unit tests? What other markets could you tap instead of fixing your integer overflows?
There's a peculiar kind of survivorship bias when looking at code. The fact that you're being paid to look at it means that it was successful enough to actually get users. Usually, that means some sort of trade-offs to actually get to market; the people who spend time writing unit tests and figuring out the perfect design get beaten by competitors who don't (trust me, this happened to my first startup). Thus, all code sucks, and all maintenance programmers grumble, and we'll continue living with shitty software because that shitty software satisfies users better than perfect software.
The fallacy here is that cleaning up your code, doing things right, and writing unit tests slows down your development speed. You could be implementing new features, after all.
But what you will (or would) find if you did take the time to do it right is that it speeds up your development speed tremendously.
I am not talking about proof of concept implementations and skeleton apps, rapid prototyping - these all have their place.
I don't want to write the next Twitter fail whale server app - thanks very much. As for Facebook, they obviously have a rock solid backbone infrastructure. Extremely impressive, and I am willing to bet that it didn't get there by adding hacks upon hacks - to the contrary.
I'm working on code that's over 10 years old (Java) and it's grown organically throughout that period, meandering through whatever technology is trendy (Servlets -> JSP -> JSF -> Dojo).
The problems have not gone away and are, in fact, getting progressively worse. On the other hand, cleaning up the code is a massive effort with little or no perceived benefit to the customer so it won't get done :(
Hm, my understanding from poorly-recalled internet surfing was that Google had a very strong culture of code reviews. But I suppose as a former (current?) Google employee you know better.
Google has a strong culture of code reviews, but that's not mutually exclusive to accepting an expedient but ugly solution and moving on. Most code reviewers are sensitive to pressure to launch; in rare occasions, you'll see something held up while a bunch of architectural decisions are redone, but usually the point of a code review is to fix obviously wrong or easily fixed issues, not to make massive changes that will block a launch.
I'm sorry to see that the owner of promosthatrock.com has let the site they set up at colindevroe.com fade away (presumably the domain expired).
It included a full archive of his email exchange(s) with Colin Devroe, chronicling the frustrating experience of paying a developer thousands of dollars and then getting the runaround for months and ultimately getting nothing for their money.
If you have the patience to use the wayback machine, give it a look. Perhaps the customer Colin ripped off will see fit to comment here as well.
[I'm a longtime HN user with a handle connected to my real name - I don't want to get into it with Colin, but I know him (he took me for a couple grand also) and it bothers me enough to see him on the HN homepage giving advice to a community I respect that I'm posting this]
It stinks that you're deciding to remain anonymous. If I "took you for a couple grand" I am very sorry about that but I can not defend myself there without knowing who you are.
That being said, the promosthatrock situation was over 7 years ago. I definitely made a mistake there and do not deny it. Again, if I wronged you then I am sorry.
From the article: “Can I take some time to clean up this code? It is horrendous.” The answer should always be yes to this question.
No, it shouldn't. This entirely depends on the long-term prospects and importance of the code. Assuming that your boss is doing his job, he might understand the trade-offs between having you doing 'code gardening' in subsystem A vs. building new functionality in subsystem B. He may even understand them better than you.
Mileage varies, of course, depending on Pointy-Hairedness.
I agree, this seems to be based on the feeling some of us have that code has to be "done right", forgetting that the code isn't there for its own sake (at least in a job setting). The bottom line is really what matters in the end, and your boss is closer to that than you are.
On the other hand, coding efficiently in the long term will likely sometimes be good for the bottom line, and you are closer to your code's efficiency than your boss. So one should make sure to understand why the boss wants to make a certain call.
I guess you could say: Make it good. Just don't make it _too good_.
Not all code needs to be super general, extensible, and re-usable. However, all code needs to be simple and elegantly designed, and unit tested. It needs to not smell.
I think the perspective I was taking in my post was that many bosses do not fully understand the inner-workings of the codebase for their own product and they also don't need to maintain it. So developers truly are the best person to help weigh the need for maintenance.
Of course, there are bosses that fully understand all of this and if you're lucky enough to work for one - awesome.
The other point here is that the new code is new code.
The old code may have been old, but maybe it has been running for a long time without problems. Maybe some of the ugliness of the old code has to do with handling bugs and corner cases that the programmer has forgotten about.
The boss may not have to maintain the codebase and maintain the code, but he/she may have to take the heat for things when someone goes and breaks code that was ALREADY WORKING with their rewrite.
"Why did you let this idiot break our product?" will be the question that the boss will be asked, in this question, and when they say "Well, I didn't know that he was making changes for no reason" someone is on an express path to getting fired.
Fundamentally it is a deeper problem. If you have to put your hand up in order to be allowed to go to the loo, you are not in an environment that respects human dignity.
If you have to ask permission in order to do the right thing then you are in an environment that is ethically deficient. Moreover, despite everybody claiming to value success, it is likely that your definition of success is significantly different from the PMs definition of success.
Their definition of success is likely to be something like "all activities marked as completed and on time", whereas your definition of success is likely to be something like "the bloody thing actually works properly".
This is why testers have such a sucky job, because they always come under pressure from the PMs to give the final sign off on something that they know isn't working properly or 'good enough'. Testers that do their job properly and stand up to the PMs are in danger of losing their jobs. This is why testers need an 'advocate' that has at least as much political clout as the PMs (if not more so - if the PM is putting pressure on the testers to sign off on a crappy product, it isn't the testers who should get sacked...)
When I—a development manager who still works with the codebase—tell my guys that I don't want them working on that particular piece of code it's precisely because a) I know things about the overall design and project that they don't, and b) I know what needs to be done next.
Yes, I want my employees asking whether they should be spending the extra time to make something subjectively better. Once the code works, we need to weigh the time it will take to clean the code up vs the time to do the next task.
Code cleanup should NEVER be a sole task, and it should probably not be a solo task if it is (and if it is, it had better be one of your two or three best programmers in the company doing it). I have told folks not to make changes NOW, because we are revisiting the code in two sprints for the next major milestone. Meeting the current milestone is more important than aesthetics, and it isn't an ethical deficiency at all.
(Hopefully this is sensible; composed on the iPad right before bed.)
Let me be clear - I think that before programmers start throwing off their shackles and asserting their 'rights' they need to build up trust with the PMs.
I do this by consistently delivering high quality code on time and reporting back to them about progress. I assure them that I'm not going to drop them in the crapper by trying to conceal a problem until it is too late, I am very up front and honest about any problems and try to give them plenty of advance warning.
In return, I expect to build trust and the coinage of the realm is autonomy. They show me the plan, and I figure out the best way to get us across the line relatively unscathed.
I think the perspective I was taking in my post was that many bosses do not fully understand the inner-workings of the codebase for their own product and they also don't need to maintain it.
Worse yet are the bosses that used to be a part of the programmers, but moved on a long time ago. They tend to fight for preserving the existing architecture of the application, even when it is clearly inadequate for the task at hand. To them, preserving their illusory understanding of the codebase is more important than allowing the application to evolve.
You shouldn't ask permission for it because on most of the time you shouldn't be doing it. It's like going to the grocery store hungry, without a shopping list.
I'm a big believer in:
1. You don't modify code without intent to change it's behavior. You're there to fix a bug, or add a feature.
2. You leave the code you better than you found it. In fact, you elevate it to your/the-team's standards du jour.
3. That's the cost of doing business. There are no multiple estimates. It's not "Well, an hour to just fix it with a hack but it really deserves a day". That's a day. Deciding on the team's values and coding standards is a group-decision. Deciding on when to apply them isn't.
"You don't modify code without intent to change it's behavior. You're there to fix a bug, or add a feature."
This makes a lot of sense. If you clean up code on anything other than a just-in-time basis, you risk fixing up something that will never need to be modified before it goes out of use.
Choosing coding standards makes a lot of sense too. If you're coding a MVP, the probability of code going out of use is higher and your standard should be lower. Your team's comfort level with modifying dirty code (how much does it demoralize them? does modifying dirty code fit with their style?) is another consideration.
I think it could make sense to vary the desired code quality depending on the personality of its primary maintainer and how likely it is to get thrown out. A corollary is that if different coders on your team have different preferred levels of code quality, you should assign them to different parts of the project on that basis.
I think there is a fair amount of room to apply microeconomics-type thinking here, along with behavioral economics-type thinking about what biases (e.g. hyperbolic discounting) might cause one to choose the wrong level of code quality. And there's also the opportunity for professional development as coders become more comfortable programming at different spots on their personal output speed/code quality potential possibility curves.
Thanks. Regarding 'dirty code' though, you've reminded me of another point I believe in.
4. All code is temporary. I said du jour for a reason. It's all wonderful for a little while, but there's no such thing as "Okay, that's been refactored now", in an lasting sense. We're building and maintaining sand castles. Either the world around the code changes, or our opinions about it do. "dirty" code doesn't exist to me, at least not in some conveniently boolean state. The half-life of a piece of code depends on a number of variables, but there's no such thing as code that doesn't start decaying the moment if leaves your keyboard.
But code can be dirtier if its implementation was poorly thought out, half of it was modified to reflect new assumptions while the other half was worked around and left intact, etc.
My algorithms professor engrained an idea in me that has stuck throughout my career: first you make it work, then you make it better. When it comes to code, we want an almost mathematically-provable correctness, but the real world doesn't care. The real world cares about getting something that solves their problem; ugliness, code smells, and technical dept are irrelevant to them. So I wouldn't write off the managers and their fancy business needs as just some conspiracy keeping good coders down. I think this need for pragmatism is something that separates a computer scientist from an engineer.
It goes both ways though. Sometimes the shit is just terrible. When the cost of any change is introducing a new bug, you've got to take a stand and I think your insight is spot on here. The developer is in the best position to know when the cost of change has gotten to high. I don't bother asking permission, but I've never had to ask forgiveness either. The manager is my customer: he just cares that what comes out of me works, and because I stop to clean up, it does.
This article is kind of the equivalent of saying if one aspirin is good, ten must be great.
The better logic for follow, when cleaning code, is the concept of technical debt (http://gettingreal.37signals.com/ch10_Manage_Debt.php or http://www.codinghorror.com/blog/2009/02/paying-down-your-te...). Essentially the concept is that quality code is an investment, and hacked code is a loan. Too many loans and you'll file bankrupty, too much time on quality and you'll not be able to build fast enough - balance is key. This idea is easy for everyone, from noob to pro to manager, to understand and establishes a common lingo.
As a manager, if I found a fresh code monkey doing nothing but assuming they had the aptitude to pay down my standing code depts, I'd promptly show him the door for grossly undervaluing the abilities of his peers.
Can anyone in the web-based software realm back up that scaling costs mythos? I get it in the hardware realm, and to a lesser extent in the distributable software realm... But having worked with everyone from Salesforce.com to Facebook to Amazon, I've never seen any indication of exponential costs from coding to testing to deployment - at worst I would say it is linear.
I am working on web projects and I can confirm the costs scale. It gets even worse if you are doing e-commerce, then you lose real $$$ as well.
You may think with all the fancy tools and smart frameworks you should be safe. But refactoring becomes more and more difficult as you incur debt and your tools and frameworks give up on you at one point.
If I recall from my Software Engineering courses this is based on research that IBM did, back when they still did research into this sort of thing.
There is nothing magical about the "web-based software realm" that separates it from other sorts of software and development and allows it to bypass the issues that cause the cost of fixing a bug to sky-rocket like this the later in the process you leave it.
"There is nothing magical about the "web-based software realm" that separates it from other sorts of software and development and allows it to bypass the issues that cause the cost of fixing a bug to sky-rocket like this the later in the process you leave it."
Sure there is - people writing easily deployed consumer software don't write code like this. It's crazy to spend months designing software, based on things you think you know, when you could just build and release it and have far more information in far less time. Anyone who writes software where the deployment cycle is measured in minutes to hours (that's anyone on the web, or desktop or mobile apps with auto-updaters) relies on an iterative process of code -> release -> gather data from real users -> refine the product -> repeat.
People who don't do this get creamed in the marketplace, because their competition does, and their competition gets far more information to build a better product through it.
Most of the stuff taught in Software Engineering courses is complete bullshit when applied to the types of problems that actual practitioners face. This is because the field of software engineering is far broader than most people can imagine, and certainly more than you can teach in an undergrad education. A development cycle like you site is pretty much the norm if you're writing embedded microcontrollers for avionics (actually, the cost increases are probably even steeper). Very few people are writing embedded microcontrollers for avionics. For the majority of us, who're working on software developed and deployed over the web, the cost curve you cite is ridiculous.
The best web developers I know are also the fastest I know; they write unit tests like crazy. They write good code.
I guess you're assuming that writing good code takes longer, which is the basic fallacy in software engineering. Writing good code is way faster, almost from the get-go. You reap the benefits on _day two_, literally.
I am not sure what they taught you in SE - I found most of what I learned to be true in the field, however, I also found it shocking how little this seemingly common knowledge is used out there.
I directly benefit from the incompetence of most software engineers out there, but I much prefer to work with good people.
The best web developers I know are also the fastest I know; they write unit tests like crazy. They write good code.
I guess you're assuming that writing good code takes longer, which is the basic fallacy in software engineering.
You keep saying this as if you believe that writing unit tests automatically means you've written good code. You can possibly have good code without unit tests. You can, more importantly, have awful code with unit tests.
For me, a major axis of "good" for code is maintainability. Therefore, writing good code means spending some time considering names rather than just using the first ones that pop into your head as you type. It means thinking about whether these superficially similar pieces of code are really related enough for code reuse to be beneficial, or whether it's only incidental similarity and really should exist in both places independently in your codebase. It's about writing small pieces of functionality yourself when otherwise you'd drop in a library that dwarfs your written code -- sure, dropping in the library is fast and easy, but if something goes wrong, you're now debugging 100K lines of someone else's code instead of the 1K lines of in-house code it would have taken to do just what you needed...
Writing good code in this way often takes considerably longer than just cranking out something that works for now, and the largest factor is not time to write unit tests, but time to think about what you're doing rather than coding at the speed you can type.
"You keep saying this as if you believe that writing unit tests automatically means you've written good code"
Actually he didn't state there was a causal link between writing unit tests and writing good code. In fact, if you look closely, you see that there is a full stop in between those two statements, they are in different sentences!
I'd happily believe that someone who writes good code can also write good unit tests.
I'd also strongly assert the negative of that. Someone who writes bad code cannot possibly write good unit tests, because unit tests are code.
Everything else you say I agree with 110%. Good names for code entities: yes, indeed. Taking some time up front and thinking about the design and how it will fit together... yes, a thousand times yes.
However, the last bit about it taking longer to do this than just cranking something out I (and I think the person to whom you are replying) would disagree with except in the most trivial cases.
If what you are doing can be done by banging out a perl one-liner, by all means do that and move on. Except that of coruse if you ever need to debug it or maintain it there is a higher cost involved. For all code, as soon as you start debugging it you start reaping the rewards of doing it properly. For non-trivial code, those costs will rapidly outweigh the initial savings of just banging something out.
Actually he didn't state there was a causal link between writing unit tests and writing good code. In fact, if you look closely, you see that there is a full stop in between those two statements, they are in different sentences!
Well, idiomatically, the second sentence is a kind of restatement of the first. My comment was really a reply to all nikster's similar comments in this page; he's talking about unit tests, and those he's talking to are mentioning good code, so it seems as though he believes that good code is essentially the same as "have unit tests", which is what I took issue with. :)
For all code, as soon as you start debugging it you start reaping the rewards of doing it properly. For non-trivial code, those costs will rapidly outweigh the initial savings of just banging something out.
I actually don't disagree with this at all. It's just that I don't think the costs will necessarily be in hours spent, though sometimes they will be. Sometimes the cost will be that you have to find someone at least as smart as the original coder, and that might be hard. It might be so hard that hiring someone to completely rewrite the system from scratch might be cheaper.
So, I'm not saying that people shouldn't take the time to write good code. I'm saying that it's possible to write working code at the speed of typing for some people on some projects, and this can be kept up for long enough for the savings to completely outweigh the other costs if the number of bugs is small enough. I wouldn't recommend depending on this being possible for a given team and project, but I've seen it done -- I've done it myself. The result wasn't what I'd call good code, but it worked and it was done in time to be sold for enough to keep the wolf from the door; if I'd spent half my coding time writing tests on that one, I would simply have failed to deliver.
"Sure there is - people writing easily deployed consumer software don't write code like this."
Like what?
"It's crazy to spend months designing software, "
Months? No one said anything about months!
You're doing the standard methodology bait-and-switch evangelism trick. Specifically, that the only alternative to $flavour_of_the_month methodology is Waterfall. Which is rubbish.
But by all means, enjoy your strawman karma.
"Most of the stuff taught in Software Engineering courses is complete bullshit"
Geez Louise, is that what set you off? The mere mention of Software Engineering? I'm sorry your experience with your software engineering course was so negative. We learned about useful things like big O notation and the mythical man month. Now I'll be the first to admit that I don't necessarily apply that stuff every day (particularly the big O stuff as I avoid premature optimisation... oh wait, that's another software engineering thing, shoot, sorry about that). I think for instance it's been almost 10 years since I had a discussion with someone about cyclomatic complexity (he wanted to twist the code in strange and bizarre ways in order to avoid it, I wanted to handle it by checking the parameters were good at the start of the function instead. I won because my way was simpler, more robust, faster to implement, and actually worked :D ). But knowledge of that stuff is useful whenever I do testing, and it informs everything I do, I have synthesized it.
Sounds like your comp sci professors sucked. Sorry about that, mine were very good (or rather, I avoided the courses of those that weren't)
The cost of fixing bugs is correlated with the value your software provides. If you write any software that not much people are using and relying on it (I mean, relying on it to do their job, manage their bank account, their customers...), then a bug on a production system will probably cost you a lot. The other side of that is that code where you don't need software engineering cannot be really relied on. Hacking will help you getting a prototype fast, but if you want to scale you need software engineering. Twitter (amongst others) learned it the hard way.
This is based on research IBM did back in the 1970s on mainframe applications written in COBOL and IBM 360 assembly. Since then, we have 30 years of experience and tools for making software more flexible and easier to change. It worked, although people like Barry Boehm seem not to have noticed yet. We now routinely change the requirements and design of software that has already been deployed.
As long as you are the only person usually working on the affected parts, sure. Just go ahead.
But if multiple people are working with this component or are calling it (in case you want to fix a broken interface), I think some discussion with your fellow team members would be in order as they will have to adapt to whatever you produce.
I agree on the general sentiment of not telling the boss though: It's hard to make non-programmers understand the burden that is ugly code. They think in features and whether something is "visible for the customer" (if it isn't, then it might as well not exist).
Such an attitude can lead you to having to make a decision when implementing a feature: "Oh - this code here is really shitty. I could now a) clean this up and cleanly add the feature or b) just hack the feature in somehow, maybe breaking encapsulation a bit more"
Bosses and people concerned about their free time think b), I tend to think a).
The problem with b) is that you get to your goal much more quickly (which the feature-oriented people like as it makes you seem more productive), but you are incurring dept. Now the bad code just got worse until at one point, something really breaks badly and then you WILL have to pay the price for cleaning up. This might very well be at a time where you are pressed by some deadline which means a lot of after-hours work for you.
Then again, in a team, the might or might not be you that has to clean up (you certainly hope it isn't you).
So doing it the quick route isn't just irresponsible to the project, it's also irresponsible to your team mates because they might be the unlucky ones the bad code breaks over.
So if possible, try to take those extra hours to do it the clean way. The product as a whole will get better and your team mates will not be pissed if they have to clean up your mess.
Why is this the typical reaction? Because bosses don’t have to read, edit and support the code.
I dunno about that -- mine actually watches SCM check-ins and flips out if there's some unauthorized code cleanup happening (it "adds risk", supposedly).
And yes, I've also been asked to remove unit tests because they were "making things break" (really just causing builds to fail if a regression was introduced).
Does your boss understand the concept of tests? Removing unit tests because they're "making things break" is like removing your smoke alarm because it's "making annoying noises."
To stretch the analogy, it's only a matter of time before the building burns down. If you can't come to a common understanding about this, you need to leave the building ASAP. Find another job.
1) Cleaning up code adds 0 value to the customer today. It may add value for the customer down the road, but see #3
2) Most engineers underestimate the time & effort to clean up code without altering it's behavior, and any alteration of its behavior is almost certain to introduce regressions and/or bugs
3) If you knew for certain that your code was still going to be used and hacked on 5-10 years from now, then it would probably be a net positive, but you can't predict that. If you think you can predict that, you're wrong.
While I agree with the concept 100% - doing this could land you in trouble if you're billing for your hours.
My approach is different (I run a consulting company): when we take over existing code base, I always talk about it upfront with the client: "we're going to take 10% of the time to clean up the code and do necessary refactorings". I then explain to them about technical debt and broken windows theory. Works about 95% of the time - and funny thing is, clients that don't agree with it, end up not working out anyway ...
I bill for hours and I'll definitely go and refactor something - but only if I need to touch the code anyway in order to implement a new feature or fix a bug.
I'll fix _every_ bug the right way, the first time; that may involve refactoring underlying design weaknesses that caused the bug in the first place. I also make sure that this or a similar bug don't happen, which also may cause refactoring.
No one's ever complained... I guess that's because it's actually very time efficient in the medium and long term, and usually also in the short term.
I object though, on general principles, about having to explain in detail to the client that you are going to do this.
If I did have to give an explanation, I would give a simpler analogy - I would say that it is like bringing a car to a mechanic and telling him to fix it. He opens the engine bay and sees that the engine is totally encrusted in mud. As part of his job, he will clean the mud off, otherwise he cannot see the engine properly, and that makes it impossible to see what is broken.
Great idea until your code clean up creates a bug that isn't in the original 'ugly' version and suddenly you've gone from not asking permission to begging forgiveness. Depending on what you're working on that might be too great a risk.
This is highly subjective. There is often much more at stake than clean code. On one end if you are shipping source code to an SDK of some sort, the clean code is extremely important, as it is documentation for your users. But, if it's an ancillary system on an internal project, well, sorry, but it probably isn't worth the time cost and risk of introducing new bugs by refactoring. Talk to your boss about it. If they can't articulate the reasons why it's not a good idea to clean up that code, then you should get a different job. Also (self promotion), see: http://www.jdconley.com/blog/archive/2009/01/26/put-down-the...
It's not just poor code which does it, it's code that has become tangled due to subsequent extensions and interactions.
Also, hopefully after a few months of project and codebase specific work, better ways to do what you did will start coming to mind - that's not the same as being a poor coder in the judgmental sense.
"Ask for forgiveness, not for permission
...
Any sufficiently large institution has something to lose...Corporate systems are optimized for saying no. Maintain the status quo. No risk of failure and a spectacular blowout...This is exactly why you are better off going ahead and doing something without asking first. If you don’t ask, no one can tell you to not do it."
I'd argue that the specifc principle "never ask permission to clean up code" can be generalised to "never ask permission", especially for software houses that have grown large and sclerotic. But YMMV; in a previous job I once told my manager I had an exciting idea for an automated test platform that I'd be willing to implement and test in my free time. His response was "And when you're finished, who's going to maintain it? I'd need to hire someone at least as skilled as you to keep it running. Please don't make it."
At my current job, I basically started cleaning up code in my spare time, just because I was leaving. But just deciding, oh wait, I'll just ignore my boss, and whatever culture is in place, on company time, is basically saying "I'm inexperienced."
Someone here recently talked about a factoring algorithm that some entry level programmer just decided had to be "cleaned up." He had to be lectured about the bits he failed to understand.
I think this depends on how experienced you are. If you are only a few years out of University, then perhaps you should go and find the original author and ask them why the code is the way it is, before you dive in to "fix" things.
On the other hand, if you've got decades of experience then you probably know what you're doing - don't waste time, just get on with it.
Be very careful starting down this road though ... especially if you don't have tests in place or are not intimately familiar with the codebase or you might find that your 4 hour sugar high turns into a nightmarish week of trying to fix random bugs from your refactoring effort.
How long the application is supposed to live, should be relevant here. Some stuff can be a piece of crap, even if it goes against some sensibility of a developer, if it isn't suppose to live long and can give you something back. Some times it should be up to the people that know how long it'll live. It's first when you make the money to pay for the refactoring, you should do it.
But hey, make it perfect from the get go ;)
I know, temporary stuff has a tendency to become permanent. But if you see this trend, then you and your manager should be grown-up enough to have the discussion of rewriting it for something more longterm. (It all depends on what sort of project it is).
Long term projects should never voluntarily allow technical debt. It's just bad for business.
Standard practice should be to refactor as necessary as part of any major code change and to always leave a code base cleaner than you found it. With that you only need a few targeted dedicated code clean-up projects.
This is a paraphrase of the advice that Martin Fowler gave in the book 'Refactoring: Improving the Design of Existing Code'. Kind of a shame no one mentioned that, but then it's been ten years.
You got a warning from my site? It is just a Wordpress site. Nothing too special going on that I know of. Let me know if you still get it and, if you know, how I can make sure it doesn't happen for everyone.
I'm very confused as to why my previous comment got downvoted. HN confuses me occasionally. Is it because I use WOT? Or because I brought attention to an issue with the site? Or "playas b hatin"?
I agree to an extent. It can be easy to fool yourself about what constitutes good code -- in the sense of making a product better or work on it easier. Sometimes bad code is better left as is. Even working totally unconstrained, I prefer not to refactor something unless I have a pressing reason in mind.
My rule of thumb is this: as a programmer and an employee, I am professionally bound to produce quality software efficiently. If I know I can complete an assignment faster (or in equal time, but leaving behind a better code base) by rewriting something, building a tool, fixing something architectural . . . I will silently do it. No point in asking permission. It's in my charter.
On the other hand, if I want to take a lot of time to rearchitect something -- an order of magnitude more than it would take to just do whatever it was that brought me there -- at that point, it's a strategic decision and management deserves to know about it.
The way I see it, management has no right to require me to produce an unprofessional product in my day to day work. And I have no right to force management to use engineering considerations only in strategic decisions.