They're afraid.

Fear: If I don't plan for all these use cases, they will be impossible! I will look foolish for not anticipating them. So let's give into that fear and over-architect just to be safe. A bit of the 'condom' argument applies: better to have it and not need it than to need it and not have it.

But the reality is that if your design doesn't match the future needs really well, you're going to have to refactor anyway. Hint: there will always be a future need you didn't anticipate! Software is a living organism that we shape and evolve over time. Shopify was a snowboard store, Youtube was a dating website, and Slack was a video game.

So my answer: relentlessly cut design features you don't need. Then relentlessly refactor your code when you discover you do need them. And don't be afraid of doing either of those things because it turns out they're both fun challenges. The best you can do is to try to ensure your design doesn't make it really hard to do anything you know or suspect you'll need in the future. Just don't start building what no one has asked for yet.

>They're afraid.

In many companies (think FAANG), engineers, especially senior engineers are incentivized/forced to show fancy design docs as part of the annual appraisal process. The more complicated the design, the more 'foresight', the better.

If it sounds kind of ridiculous (TPS reports from Office Space anyone?) it is. But on the other hand, taking a bit less cynical look, the more massive a company gets, the more the voices which demand objectivity in all these promotion/bonus multiplier processes. So a kind of obsession about such weird 'measurable' metrics gradually builds up in the name of objectivity.

And as soon as there are metrics, you can bet everyone in the system will do their best to game them (it only makes too much sense to do so).

And thus you end up with over engineered systems all over the place.

A lot of the 'not-invented-here, let's reinvent it' style culture also develops similarly - you have too many smart people in a room where the work is just not that demanding. Even if you were to get over your personal existential crisis (why am I writing yet another crud app?!), if you're the type that wants to see a promotion every other year, your're forced to invent work this way.

I think some problems are just seen as sexier than others.

I just wanted to shed a light on some unfortunate external pressures faced by a subset of engineers, which contributes to this problem. Didn't meant to come across as cynical.

My solution was to work someplace that can’t afford to waste money on bullshit. Has worked well so far.

It's a great concept, and is often what irritates people. I work at a national lab, and senior management is absolutely obsessed with metrics, often to the detriment of the purpose of the organization itself.

There needs to be some humility too - especially in our modern data driven world. All that is important, you may not be able to measure, and all that is measurable may not be important.

Certainly not in Amazon. Surely there can be exceptions but in general the company has a culture of simplifying stuff anywhere is possible.

IMHO the largest contributor to over engineering in this company is people suggesting flaws in other's designs simply to have something to contribute during a meeting. I can't remember anyone, ever, telling me to remove something from a design doc (3.5 years).

I have added unnecessary complexity to my own designs as a response to comments. Not customer driven, not data driven, but somebody at a meeting got focused on something and it ended up getting added to the design.

I've had unnecessary complexity introduced to my code during code reviews as a response to comments, too. Most often it's to make things "more testable" so as to reach an arbitrary code coverage target.

https://raw.githubusercontent.com/aws-samples/aws-refarch-wo...

Ironically, that's how you are supposed to run Wordpress on AWS.

My original comment was 100% based on my experience as a developer at Amazon.

Most of the time it's people from the team that is hiring and one or two "guests" from other teams (but usually working in the same building).

Managers also have plenty of power to influence the decision, therefore keeping a very uniform hiring bar is really difficult. (But no, it's not "completely arbitrary")

Also, I wrote that the bar dropped a lot, to the point of shifting all employees level up by one, but this does not mean that the company hires 90% of the candidates.

If a team was hiring 1 candidate every 1000 screened resumes and now it's 1 in 100 it's a whopping 10x change... but that doesn't make you a terrible engineer!

Say one slightly dubious thing and you torpedo your chances of the job. Write one slightly dubious thing on your CV, and it will be rejected without a second look.

While I’m confident that most of the people rejected for those reasons would be at least adequate at their job.

Comprehensive Unit/Integration tests, a robust type-system, pick whatever suits your style.

It's a lot easier to refactor stuff when you don't have to worry about breaking something hard to debug with a big code change.

I started out by writing unit tests for everything, which became my handhold and documentation for what the system originally did. Then I started reorganising the code into a more structured and more readable form, without changing the functionality, as proven by my unit tests. Then I started asking domain experts what exactly it should do, unit tests in hand, asking if these answers were correct. If they weren't, I changed the unit test, and then changed the code to match.

Surprisingly painless for something that by all reasonable standards was a terrible mess.

Great book BTW, should be on a top10 must read list for software developers. (#1 will always be Peopleware[2])

[1] https://www.goodreads.com/book/show/44919.Working_Effectivel... [2] https://www.goodreads.com/book/show/67825.Peopleware

Our CIO was regaling us with the story about how they were going to change our office today, make it more open, fancy like Google or Facebook, free desk. But when I asked if they’d considered dealing with the noise issues we were having, no, no they hadn’t considered that.

It just blows my mind. I just really wanted to ask them what the hell they thought they were doing modifying the office layout without asking the people who need to work there.

I also don't see how a robust type really helps there. It might even be part of the thing that needs to be refactored. Besides, many languages don't have a very robust type system.

End-to-end tests and type systems certainly have their uses, but for refactoring messy code, I don't think there's a good substitute for thorough unit tests.

Although there are different kinda of refactoring of course. The case I'm referring to was about one very messy module. This makes it very easy to unit test. If instead it's the entire architecture of your application that needs to be refactored, then you're looking at a very different case, and end-to-end tests become more important than edge cases.

End-to-end (e2e) tests the entire stack: deploy the site with database and all, run a script that visits a page, clicks and stuff, and checks if the right things become visible.

Unit tests are the other extreme: you take a single unit of code and test whether it does what it's supposed to, while mocking all communication with other units of code.

Integration tests are in between those two extremes, and probably the least understood as a result (at least by me). They look like unit tests, and don't generally test the UI, but they don't mock the other components, and ideally also set up a database to test against.

I think there's a bit of overlap between unit and integration tests; a sloppy unit test where you don't mock (some) other components but treat them as part of the unit you're testing, start to look like integration tests at some point. If you want a clear demarcation, I think you might consider a database connection essential to count as integration test.

In Ada, the programmer is discouraged from using the Ada equivalent of int directly, and is encouraged to instead introduce a subtype that reflects the specific use of int (including automatic range checking).

This isn't as natural in C++ but is still possible. Boost offers a BOOST_STRONG_TYPEDEF [0] to deliberately introduce an incompatible type. (I do recall having trouble getting it to behave, but it's been a while.)

Whether this makes sense in most mathematical code, I'm not sure, but it seems like it's an option.

[0] https://www.boost.org/doc/libs/1_73_0/boost/serialization/st...

[0] https://github.com/nholthaus/units

[1] https://github.com/mpusz/units

For an example, dig into some crypto libraries. They operate on bytes all over the place performing XORs, etc. A type system isn’t really gonna help you ensure you got the correct number of AES rounds and stitched the blocks in the right order.

IMO the only systems where this “type system eliminates most tests” philosophy seriously works are the ones that don’t do anything other than pass data between components without doing anything beyond calling some serialization methods.

Which is what 90% of programmers on this website are essentially doing. And for the remaining 10% there is likely a better suited, different programming language or type system available. Even if there isn't unit tests would still be very niche and the general case would be that by default you shouldn't be unit testing.

End to end tests are really slow, but if you can get them into the 300-1600 tests per second range then i have no beef. I value tests but I seriously grudge waiting for tests.

Also, have you ever seen a project where the code is a mess but the unit tests are perfect? Even if somehow you could write unit tests that would cover for super bad code (which you can't), it is extremely unlikely that your unit tests would be that amazing.

There's quite a lot you can refactor without breaking unit tests. If you've got a single 200-line function full of nested loops with cryptic variable names, modern IDEs make it really easy to extract those loops to their own functions. Figure out what they're meant to do, give them a descriptive name, and you already improved the code a bit without breaking any unit tests. If your IDE does this well, you could even do this without unit tests, but you really will need those unit tests once you start reorganising the code making use of the excessive number of parameters those extracted functions invariably end up with.

Of course you can write unit tests for super bad code. If it's a function that returns something, it's trivial to unit test, no matter how badly written that function is. If it calls other code, you have to mock those calls and test that those mocks get called under the same conditions. If they mess with global variables, that's terrible, but even that can be mocked.

If the code uses gotos to code outside the module, somebody needs to get shot, and I guess you need a unit testing framework that can mock those gotos. I've never seen one, because nobodoy uses gotos anymore.

With end-to-end tests such as when piloting a browser, it's not really easy to get things like tracebacks into the console output for example.

If you're working on a very specific box that has well defined, well known inputs and outputs then TDD is an excellent tool.

But for anything with a non-specific "We'd like to do X and display the result on Y" it just gets in the way.

I've worked on mobile apps before with a small team, and inevitably, we'll find bugs that show up in the user interface when the user rotates their phone. It's hard to unit test for rotation changes, and it's also hard to code a rotation change into an end-to-end test on mobile. Animations are also something that's difficult to test in an automated fashion, and all the testing in the world won't be as good as showing the animation in front of a designer. So some level of manual testing is needed on mobile.

I've worked on web frontends where there would be bugs with scrolling jumping back and forth. An end-to-end test using Selenium may not catch the issue, but for a user, it can be painfully obvious. Similarly, animations are also hard to unit test on the web. So some level of manual testing is needed on the web.

The only place where I could see manual testing NOT being needed is for backend development, since the input and output to a backend system is much more controlled. You could write an end-to-end test for any scenario a user could throw at your system.

In summary, don't underestimate the value of manual QA!

Seems like that "refactoring untested code" would be a well established recipe for disaster.

Not sure if it counts as a 'way', but a static type system helps too.

I have noticed people who are extra organized in real life, who keeps every single file in a right directories after download tend to have inclination for prematured code refactoring for future use.

If these guys become code architect then i end up doing so many unnecessary things. The fundamental assumption of the refactoring gets changed very fast and the code needs to be rewritten for the majority of the cases.

From a company level I find the instructions are clear, mostly holding the same contract between services as long as possible and less schema change.

Its the engineer with subjective idea of perfect code / supporting future work makes it even more complex

As the GP says, this is about fear. Fear that if you don't plan for it now, you won't be able to deal with it if it happens. Or in architecture terms, fear that your system will be faced with a requirement that it can't cover.

Trying to plan for every eventuality is usually wasted effort. Better to build robustly, and trust in your ability to adapt.

Don't do this and solve only for the problems you have now or are about to start in the next 2 sprints.

edit: and make refactoring acceptable and part of your engineering culture so you do it often

This is knowledge most engineers I've met completely ignore in favour of the superstitions, personal opinions, and catchy slogans that came out of the '90s and '00s. It's common to dismiss the early software engineering approaches with "waterfall does not work" – as if the people in the '60s didn't already know that?! Rest assured, the published software engineers of the '60s were as strong proponents of agile as anyone is today.

Read this early stuff.

Read the reports on the NATO software engineering conferences.

Read the papers by David Parnas on software modularity and designing for extension and contraction.

Read more written by Ward Cunningham, Alan Perlis, Edsger Dijkstra, Douglas McIlroy, Brian Randell, Peter Naur.

To some extent, we already know how to write software well. There's just nobody teaching this knowledge – you have to seek it yourself.

You can read all the papers you want. At the end of the day, you actually need to practise writing code! Every piece of software involves a different domain, different requirements, etc.

I feel like the majority of developers these days just copy and paste stuff from the internet and glue some npm/nuget/maven packages together, brush their hands together and feel like gods. That is not how you become a good developer!

How do you become a good developer? Write code _yourself_. Then rewrite it again and again. Keep thinking "how can this be done better, cleaner, more elegantly? how could this be more readable, maintainable?" .. "Maybe others have found a better solution for this, let's do some research and investigate all options, weigh the pros and cons, and select the solution that fits best for this particular problem". Don't forget to factor in the cost of complexity. Is the code you are writing too difficult to understand for the next developer? Is the abstraction too rigid and complex? There is a cost to that.

Rinse and repeat. Sooner or later, once you've written enough code and thought critically about every line, you will acquire a "feel" for what it right and what is wrong. Then suddenly you are an industry expert. Suddenly you are training and mentoring others. Because you put in the goddamn effort.

Even then I think you have to be able to look at your own code critically and want to make it better. The current tech lead in my company churns out a lot of code quickly, but doesn't take constructive criticism of it at all well. It's not easy code for other people to jump into and there are no docs or tests. He thinks it's close to perfect. I don't

I've been disappointed with the industry for the opposite reason: I don't want to be doing this, but this is what is being asked of me.

A current favorite of mine is "The Emperor's old cloths", the Turing award lecture given by C.A.R Hoare. In particular his line "The price of reliability is the pursuit of the utmost simplicity", it applies equally to maintainability and extensibility (I think these are part of what it means for a system to be reliable).

An idea I try to keep in mind while working is not to plan or build for future features but simply leave room for them (meaning don't actively prevent their eventual existence through complexity). It has taken some practice, but it helps guide to a simpler implementation.

Frequently forgotten is the duality of extensibility: subsettability, or contraction. Being able to remove or disable code without rewriting large parts of the application is just as important as being able to extend it!

Especially Microsoft has a tendency to throw out frameworks and helpers that just don’t provide the needed flexibility, and once that has been addressed what’s left is a heap of extensibility points that wrap three lines of useful logic in hundreds of lines of framework code.

So if I could ask one thing of platform developers it is to leave the framework design to the user, and just provide useful primitives. The two hours spent on writing the extra glue code is easily saved on the days spent trying to learn the framework.

In answering the original question, one would hope that focusing on the risks to successful completion should de-prioritize the fixing of things that are unlikely to become problems in practice.

https://dl.acm.org/doi/10.1145/12944.12948

It even lead to Stroustroup saying in a talk that one shouldnt use linked list at all, but instead vectors, because even when linked lists should shine, they do not anymore.

[1]: https://en.wikipedia.org/wiki/Design_Patterns

Good abstractions at the correct level and the correct data modelling can make the code around it very simple. Instead of nice designs that are easy to reason about, we have lots of unit tests to "prove" that it works. Instead of making a system reliable and trap errors properly, we use Kubernetes to launch another pods as soon as one goes down. Maybe I am just too old, but I feel we are loosing something going down this route.

(Ok, there is code that's functionality will mean that in inherently isn't simple, but I see far more CRUD apps that are built in an over complicatd manner, than code that is actually doing something algorithmically complex).

Writing perfect bug-free code is nearly impossible and expensive. Hardware is going to fail, network links can get flakey, packets can get lost. Accepting that failure is going to happen no matter what and building for resiliency to it is a far better approach.

Unit tests are along the same line- its been measured that fixing a bug in production is 10x more expensive than finding it in development. Unit tests help ensure that more bugs are found in development, there is less need for, slow, expensive, and error-prone manual QA processes that hold up release frequency.

Unit tests also help document the code to some extent. As teams turnover and new developers who may not fully understand the nuances of some code come in, Unit tests create guard rails that help catch edge cases that may not be obvious to new members of the team and when the test fails makes you go back and give more thought to those areas and why they behave the way they do.

Rereading this, you claim to be on the older side, which I am surprised about- this seems to me like an opinion that someone who is relatively new to the industry may have. I am continually amazed in the new and creative ways things can fail, building for resilience is a 100x better approach that also reduces stress tremendously- Instead of "All hands on deck no one is going home tonight- we just took the system down with this last release and will need to patch it ASAP" the conversation goes to "Ok- we rolled out the release to 1% of our users and we see error rates spiking. Lets roll it back and look at the logs to see what happened."

Similarly with unit tests- it used to be monthly or weekly release cycles, now we are down to multiple times a day because I don't need a programmer who washed out to get around to clicking on the right buttons to give it a green light.

These are tremendous improvements in the state of the art.

> This is about cost optimization to some extent.

Is all the complexity and developer time that is required with Kubernetes really cost effective? We added it at my work and it took around 6 months of developer time from what I could tell (it wasn't me who set it up). Now we have to debug it and none of our team is an expert, so lots of guesswork and googling. Using number of small pods has given us a lot more problems than one big machine. If you are running out of memory because of a large upload, restarting a pod when it crashes doesn't help you.

Kubernetes has lowered our cloud costs and given us high availability, but it has increased development time in a number of ways. Maybe as we get more experience it will get better, but so far I would not say costs have been optimized as a result.

If anything, I think the Greenfield nature may have more to do with it. Today, we take it for granted that software is really expensive to make, to the point where we don't always question the basic patterns that make it so expensive. In those early days, it was far from obvious that software would be expensive.

The mid-career developer knows this from experience and sticks to the minimum necessary to meet the requirements. Fast and efficient in the short term.

The senior developer mostly agrees with that but also knows that there is nothing new under the sun and all software ideas repeat. So from experience they can selectively pick the additional implementation work to be done ahead of time because it'll save a lot later even if it's not needed yet.

As an aside, this is one of the many reasons why I interview based on discussing past projects and don't care for algorithm puzzles. Unless I specifically need an entry-level developer, I'd prefer to have the person who has written a few silly (in hindsight) complex frameworks and has painted themselves into a corner a few times with overly simplistic initial implementations. That's the person I know I can leave alone and they'll make sane decisions without any supervision. The algorithm puzzle jockey, not so much.

The mid-career is perfectly fine, give you are also good at refactoring if needed.

Also, another thing that I'm not seeing stressed enough on this topic is: measure.

If you're writing code for a device that's going to hang out in the forest for 10 years with no updates, write just enough code to solve the problem and make it easy to test. Then test the fsck out of it.

If you're writing code for a CRUD web service that you know will get rewritten in 10 months, write just enough code to solve the problem and make it easy to test. Then, test the fsck out of it.

If you're writing an Enterprise app that will be expanded over the next half decade, write just enough code to solve the problem and make it easy to test. Then, test the fsck out of it. You simply cannot know how your code will have to change so you cannot design an architecture to make any arbitrary change easy. Accept that. The best you can do is write clean, testable code. Anyone... anyone who thinks they can design a system that can handle any change is flat wrong. And yet... time and again, Architecture Astronauts try it.

Nonsense.

Of course you can make reasoned predictions about how your software will change or not in the future.

Some will be wrong, but if you have any sense many will be right enough to have been worthwhile to address up front, and the balance will be clearly positive.

E.g.:

def current_time

  Time.now

vs. just repeating Time.now inline within all of the calling methods?

When it happens, deal with it - odds are it won't. Expend the energy making the code base testable not pretending to be an insurance company ("In case sh..").

But I think it's silly to imagine that there's nothing you can do to anticipate changes and make your code more amenable to change.

This is where "make it easy to test" comes in.

And, sure, there are some things that are easy to anticipate. If you're processing CSV, you may get asked to process arbitrary delimited files next. But the point is 3 - 5 years out, you'll get asked for something the current architecture cannot support and a major effort will be required. You cannot anticipate those and thus should never try to.

A junior engineer recently presented me with some completed work. There was a data schema change on one interface and an API version change on another. He had created a version of the microservice he was working on which could be configured to work on either the old or new version of each of these, with feature flags, switching functionality and maintaining the old and new model hierarchies for both. He viewed this as an achievement. While technically it was, the result was code bloat and unnecessary complexity. The API upgrade and schema change were happening at the same time, and would never be rolled back, his customisability was a net negative.

Code is a liability. Be sparse. Build with some flexibility but overall just build what is required and when the future comes, change with it. Don't build to requirements you can imagine, because the ones you don't will kick your ass.

If you need to push every variable through 15 layers of interfaces and factories and generators, you'll have a bad time.

Atomically? What if your DDL times out?

What you describe is, in my experience, extremely standard process for rolling out breaking changes (you do of course remove the switch and old version support after everything is rolled out).

Effectively yes, during a defined outage period (don't ask, this is finance), on any error anywhere in the platform during rollout, the entire platform would be reset to its prior state.

> What you describe is, in my experience, extremely standard process for rolling out breaking changes

But entirely unnecessary here. There was no requirement for a single version of the microservice to be able to address multiple versions of either dependency. It was a net negative to have that support, added significantly to the software complexity and the raw LoC.

But this is hard to do. I once made a major feature change - where I feared the consequences and bugtracking afterwards - but it worked like a charme, because the complex abstractions I made in the beginning, really helped, so it was done in a couple of days and not weeks or months, like I thought. I was really proud of my older me, that day.

Only, like you said, when you have layer of layer of abstractions, the problems rise. When you really have to dig in to understand what is going on. Or when the abstractions are simply not helpful for the new change, which happens way too often, too.

"Code is a liability. Be sparse. Build with some flexibility but overall just build what is required and when the future comes, change with it"

So yes, I very much agree to that.

Period.

Instead, only build what you need to solve the problems you have in-hand, but do so in such a way that your software and your systems are as easy to change as possible in the future.

Because you very, very rarely know what your future use-cases really are. Markets change over time, and your understanding of the market will also shift, both because some of your assumptions were wrong, and because you can never really have a total understanding of such a complex phenomenon.

Your business needs will change as well; your top priority from 2019 is likely very different than it is today.

That is why you build to embrace change, rather than barricade against it.

As to how, I'd start with Sandi Metz's 99 Bottles of OOP: https://www.sandimetz.com/99bottles

Learning to write readable code is also pretty important; Clean Code is a good starting point (https://amzn.to/3168z3A), but I'd be keen to know of any shorter books that cover the same materials.

Growing Object-Oriented Software Guided By Tests (GOOSGT) is a good read as well: https://amzn.to/3du1sEL

The new requirements you will need to implement are probably unknown right now.

If your code has no extra fat, it will be easier to adapt.

If you learn anything by having super complex software that tried to anticipate future needs is that you rarely get it right.

Write the code for the problem you have today. Pay attention that, by the choices you make, you don’t walk yourself into a corner.

Over-engineering can be avoided by carefully sticking to a budget.

For a lot of developers, there's a trade-off between rationality (in the sense of ROI) and feeling good. It doesn't feel good to make every engineering decision against a budget. My dopamine levels [0] skyrocket when I visualize making some component generic, or future proof. I come crashing back to earth when I realize the budget is for a driveway. Budgeting earns the bosses / customer / capital a better return. Engineering for future use cases or making things generic (or the mere anticipation thereof) is an easy way to get a massive hit of neurotransmitter that makes you feel good.

[0] Not a neuroscientist. But I think it's useful to label that spike of feel good and motivation. It may have nothing to do with dopamine.

I think I've successfully rewired that impulse in myself now, at least partially. I do get that feeling from imagining great systems, but I also get that dopamine hit from deleting old stuff and simplifying things as much as possible. We can ditch compatibility for API v8? Awesome, I just made my code smaller and we can scrap that entire abstraction layer! The service just lost 20% of its LoC :)

If you're writing foundational code for an operating system, then yeah, whatever you write is going to stick around for a while. (Which means you need to be a LOT more careful about what you write.)

If you're writing code for a mobile app or website, expect your code to get thrown out in a few months due to shifting requirements. I'm sure the Facebook app has been completely rewritten more than once, since they switched to React Native at one point.

If you're writing a JavaScript library, expect your code to be replaced next week. :D

As you learn to code, it becomes increasingly satisfying to solve problems, so you write more code, which provides more satisfaction, but it gets unwieldy quickly and you start to get lost in the spaghetti.

As you learn to abstract, it becomes increasingly satisfying to create abstractions and manipulate them afterwards, allowing you to not get lost thanks to the now beautiful mental map of your code. But at some point in time it leads to excessive abstractions and issues start to pile up, and you can't quite make sense of the map anymore because everything is a little bit too generic and doesn't carry any more meaning, and such abstractions invariably leak through the lasagna. You usually start by writing more abstractions, but this only feeds the loop.

And finally, as you learn to delete code, you get back to simplicity, and the disappearance of the mental and causality weight associated with that deleted code becomes increasingly satisfying. You start inlining code, you start repeating code, only for the repeating patterns to emerge by themselves, you start thinking about names for those, and the names come naturally. Instead of powering through, there is a state of continuous, unstoppable flow based on a loop of implementation and emergence, where a giant Rubik's cube effortlessly solves itself by throwing it in the air and falling into place.

The scope of the project then expands to fill the budget.

Running Costs + (Weekly cost of Engineers * Weeks Slated) + (Hourly Cost of Managers * Hours in Meetings/Discussions) = Projected cost / required budget.

Increasing scope causes an increase in time spent both in meetings arguing about it and engineers working to build it, which increases the time side. Increasing salaries / quality of hires increases the cost side. Promises to investors or contractual obligations to partners/customers are non-negotiable increases in scope, so directly drive how much time something takes.

Software engineering has the potential to earn a lot of money with relatively little effort.

This is both a blessing and a curse: if I just do anything that I feel like, maybe only 5% of what I do would actually carry a profit – but that profit would be big enough to compensate for the other 95% of the time when I only do things that make me feel good without getting any real return from it.

If your goal is to produce maximum value, you have to think very long and hard about what projects you take on. It is worth taking the time to consider projects and research their viability, because very many of them won't pay off in the long run. You'll have to get used to saying "no" to anything that doesn't have an obvious route to profit.

On the other hand, if your goal is to have a little fun along the way and not only to produce maximum value, you can use this fact to your advantage! If you know you've rejected a bunch of low-value projects recently, you can afford to take on some low-value (but fun!) projects and still sit firm in the knowledge that you're still producing net value in total. Because the 40% of projects you've done that do produce a lot of value will easily pay for the other 60% that aren't obviously profitable. Or whatever ratio you settle at.

Better is to design for change. Keep everything modular. Keep your concerns separated. When something needs to change, you can just change that thing. When the whole basis of the system needs to change, you can still keep all the components that don't need to change. The only future use case you should design for is change, because that's the only thing you can be certain of.

So don't make things more generic than you need today, but make sure it can be made more generic in the future. And in that future, don't just add new features all over the place, but reassess your design, and look if there's a more generic way to do the thing you're adding.

I do this sort of stuff all the time on my current project, and it works quite well. There's no part of the system we didn't rewrite at some point, but all of them were easy to rewrite.

“An architect's first work is apt to be spare and clean. He knows he doesn't know what he's doing, so he does it carefully and with great restraint.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used "next time." Sooner or later the first system is finished, and the architect, with firm, confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.”

The overall advice is to practice self-discipline:

“How does the architect avoid the second-system effect? Well, obviously he can't skip his second system. But he can be conscious of the peculiar hazards of that system, and exert extra self-discipline to avoid functional ornamentation and to avoid extrapolation of functions that are obviated by changes in assumptions and purposes.

A discipline that will open an architect's eyes is to assign each little function a value: capability x is worth not more than m bytes of memory and n microseconds per invocation. These values will guide initial decisions and serve during implementation as a guide and warning to all.

How does the project manager avoid the second-system effect? By insisting on a senior architect who has at least two systems under his belt. Too, by staying aware of the special temptations, he can ask the right questions to ensure that the philosophical concepts and objectives are fully reflected in the detailed design.”

I agree, contriving baroque features and APIs are usually wasted time. But planning simple basic operations that others are built upon is good sense. When possible.

The most important reason for us to do this is so that our customers can always accomplish what they need, even if it's somewhat manual or painful. By exposing our building blocks we never have an emergency where we need to implement the feature "copy foo to bar so that customer X can accomplish important thing Y." Instead we can say "you're able to do that manually and we'll talk about a feature that will help you automate if the manual process is too painful."

Sometimes you just need to write some straight business logic, don't get me wrong, and it takes experience to know when to create a new concept and when to just bang out some code. But exposing primitives or building blocks reduces the amount of "critical" development substantially.

but I have problem sharing it with people because I just don't the credential.

Is there a well-known book/resource that I can share with people about this?

So while you can still draw parallels, just realize that this wasn't written about what we call architects today.

By refactoring what I mean is continually revisiting the architecture of your code, identifying common functionality, better organisation of code, better abstractions. The right decisions for your codebase change as it evolves and so you need to keep reexamining these (implicit) decisions.

Continuously incrementally refactoring your code enables so many things to work better. Your example here is a great one - don't implement something you don't have a direct need for. Don't design for it. If you have a culture of refactoring then you can be confident that in the future if that need crops up, you can implement it with appropriate refactoring that the result will look at least as good as if you designed it in up front. If you don't refactor as a team, then you have to do it now, or putting it in later will result in a mess.

Code should be treated as expendable and transient, your clever implementation might be considered deprecated in a month due to changes in requirements/design.

Unfortunately it's not easy to convey this idea as a team lead. People get "attached" to their code and the idea of refactoring is viewed as mostly unnecessary and dangerous (I wrote the code, I'm not gonna rewrite it)

I had assumed in my naivety that as I got more experienced was that I would get better at doing things right 'first time' and wouldn't need to keep going back and fiddling with code I'd already written because it now wasn't quite right. What I learned instead was that he spent at least 50% of his time re-working existing code.

There are two important things. The right design changes as you add more code. You learn more about what the right design should be as your work on the code.

You're never going to get it right first time. So instead concentrate on continuously making it better.

Even then, I don't dig too deeply into things. I look for two things:

* What a migration path to realistically expected features _might_ look like. If I'm debating between two column types or table setups, go with the more flexible option.

* What scenarios will be extremely hard to get out of. Avoid those when reasonable.

----

In other words, instead of proactively planning for some feature to evolve into the unknown in some way. I'm looking to make sure I'm keeping flexibility and upgradability.

An example of this conversation might look something like "This design is nice and simple but here are the possible scaling bottlenecks. If we need to scale X then what would be the path for that?" or "There's no reason to make this configurable now but if we needed to in the future how would we do that?"

Those sorts of questions shouldn't drive the design but if you're trying to decide between two designs that feel roughly as good as each other then it's a good way to tip the scale. I also find it a good way to introduce junior devs to thinking through those kinds of ideas.

One thing I had to learn to be careful with was to make sure it's really clear we're talking about future hypotheticals. I've had cases where the conversation about future design ideas ended with "Ok I'll go get started on that" and then we have to talk about how we weren't discussing real implementation changes, just hypotheticals about how the design might work in the future.

Building something “generic” to anticipate “future use cases” is not design; it’s the postponement of design.

If you anticipate some unpredictable future feature, then either you do not understand the design space yet, or you are following the directive of a business which does not understand it.

Startups (prior to product/market fit) have a legit reason for not understanding their design space; they’re still exploring it. That makes design pretty hard. Instead of creating The One Generic System To Rule Them All, I’d recommend small, less-generic, low-risk prototypes, that can be easily replaced or refactored. Keep them uncoupled. Meanwhile, use those prototypes to build up your understanding, so that you will be able to make informed design choices at a more mature stage.

I’m speaking in broad strokes; reality may apply.

My rule of thumb for this is to repeat yourself at least 3 times before trying to build an abstraction to DRY them up.

3 use cases is obviously not always sufficient inform the design of a good abstraction, but IME:

- abstracting at 2 use cases is very often premature and results in leaky/brittle abstractions where the harm from added coupling between fundamentally incompatible usages far outweighs whatever little harm can be introduced by keeping the 2 usages as repeated code, and

- with any more than 3 use cases, the maintenance cost of keeping the usages in sync starts to scale non-linearly, so at the very least thinking about a design for a potential abstraction at that point becomes a worthwhile exercise, even if we end up deciding it's not yet appropriate (hence the "_at least_ 3 times").

Also known as WET (Write Everything Twice).

In 2017 I started https://wordsandbuttons.online/ as an experiment in unchitecture. There are no generic things there whatsoever. There are no dependencies, no libraries, no shared code. Every page is self-sustaining, it contains its own JS and CSS, and when I want to reuse some interactive widget or some piece of visualization somewhere else, I copy-paste it or rewrite it from scratch. In rare cases when I want multiple changes, I write a script that does repetitive work for me.

This feels wrong, and I'm still waiting when it'll blow up. But so far it doesn't.

Sure, it's a relatively small project, it has about 100 K lines along with the drafts and experiments. It is although inherently modular since it consists of independent pages. But still, this kind of design (or un-design) brings more benefits I could ever hope for.

Since I only code that I want on a page, every one of my pages is smaller than 64KB. Even with animations and interactive widgets.

Since all my pages are small, I have no trouble renovating my old code. Since there is no consistent design, I forget my own code all the time, but 64 KB is small enough to reread anew.

And since there are no dependencies, even within the project, I feel very confident experimenting with visuals. Worst case scenario - I'm breaking a page I'm currently working on. Happens all the time, takes minutes to fix.

I still believe in the golden middle, of course, I'll never choose the same approach in my contract work; but my faith is slowly drifting away from "design for the future" in general and "making things generic" specifically. So far it seems that keeping things simple and adoptable for the future is slightly more effective than designing them future-ready from the start.

IMO, good future-proof design is about putting in place good components and system boundaries.

Those components and boundaries can be highly specialised and have as few options as possible - it's much easier to make a system boundary more complex than to make it simpler. So start as simple as possible!

Now, with those boundaries, you can easily write tests, and iterate on the different parts of the system. Bad code in one component doesn't "infect" bad code in another part of the system.

Most "balls of mess" systems that I have seen came down to not having clear boundaries between components of the system, rather than being too generic or not generic enough.

When you cannot predict the future, flexible is often more efficient than general.

2. Understand it's very often easier, faster and better to write something trivially but twice than writing it well once;

3. Realize no matter how hard you try, unless this is something you've already done once (which brings you back to #2 above), then your information of the problem is incomplete and therefore any all-encompassing solution you may have will essentially be partial and based on extrapolation of your knowledge and guesswork.

The above also deals with a significant cause of procrastination - an internal subconscious feeling we can't do what we're about to do well, therefore we'd rather not even start it. Just sit down and tell yourself, "today I'm going to write a bad module X", and with enough time you'll be able to sit down again and write a good X.

The one thing you should spend a bit more time on is interfaces and decoupling, but as for internal implementations of things, or even entire system blocks that you can swap later on - don't bother too much. It'll be OK at the end, and you'll get there faster. Even if you need to rewrite the whole god damn thing.

Also:

1. cut features and enjoy saying no.

2. set deadlines and ship (if not enough time, see 1 above)

3. don't skip tests; makes refactoring/rewriting a breeze

1. Keep It Super Simple (KISS).

Implement the solution which works the easiest first. Copy-pasting code is ok at this point. So if an "if" will do it, use an "if". (Don't start with a BaseClass with an empty default method and a specialized class which overrides it).

Once you have something working (hopefully with some automated tests?) you are allowed to refactor and abstract. But see next point.

2. The Power of Three!

You are only allowed to abstract code once you have copy-pasted it in at least three places. If you only have two, you must resist the urge and move on. Maybe add a comment saying "this is very similar to this other part, consider abstracting if a new case appears".

After abstracting stuff, run tests again (even if they are manual) to make sure you have not broken anything.

Be warned that this method is not guaranteed to produce the "best possible code for future you". If you keep doing this long enough, you might get stuck at "local maxima" in design. Future you might need to do big refactors. Or not. That is the nature of programming IMHO.

I would look to senior people who have boots-on-the-ground experience delivering and maintaining projects/products, ask them if they think you're overdoing it.

edit: I want to add, get input from non-engineers too. Ask them "in your experience, how far have projects diverged from the initial requirements/purpose? I'm trying to plan for the future but not over-complicate things"

If I had to boil the first part down, I'd say you need to focus your engineering on the interface points - network protocols, schemas, API sets, and persistence formats immediately come to mind. It's a truism of the profession that those are the most expensive aspects of a system to change, and therefore they're the least likely to be mutable down the road.

That's really the easy part... the harder part is maintaining the team and self discipline to keep things simple. For better or for worse, engineering job placement (and oftentimes personal satisfaction) is highly resume-keyword driven. Organizations and people all tend to chase the next resume keyword on the assumption that it'll help them deliver more efficiently or get the next job placement or write the next blog post or whatever else. The net of this is that there's a very strong built in tendency for projects to veer in the direction of adding components, even before considering whether or not they're appropriate for use. So keeping your eye on actual, real project requirements over all else is both important, and can require convincing and political work throughout a team and organization.

That said, I think there are two steps in designing abstractions:

The first is to split up and isolate both code and data into small, simple pieces. These can easily be non-DRY (structurally) and often are. You'll have cases where you often (always?) pass things or do things in sequence in the same way.

The second is to merge the pieces with parametrization (or DI in OO), defining common interfaces, polymorphism etc.

The first part is very often beneficial and makes code more malleable, which makes future features/changes less cumbersome.

The second part however is the dangerous but also powerful one. It can lead to code that is much more declarative, high-level and adding new features becomes faster. But the danger is that a project isn't at the stage where these abstractions are understood well enough, so you end up trying to break them up too often.

I try to default to writing dumb, simple small pieces and deferring abstractions until they become provably apparent. Fighting the urge to refactor into DRY code.

Now there is also another issue with writing "future proof" code, which doesn't involve abstractions but rather defensive programming, which is an entirely different issue.

The first being pub-sub, this can be an event system or and observer pattern but it is good to have loose coupling of when something happens it can loosely tell it out into the ether and not care about if anyone is listening.

The second being sole responsibility of change, this does not have to be something as formal as a Redux or some other library if you are using a back-end language but there should be one function that changes one segment of data and that is the one function that owns the writing of that data.

The third is more optional but in certain places it really shines and has become kind of a lost art and that is plug-in style interfaces. Say you have a portion of you app that parses text files and stores them in a common data structure and you know that there will be future text file formats but you do not know what they will be. A plugin architecture is a natural fit here, where you define the interface and the data storage side, then you only need to implement a plugin for the parser side for each new file format. I always write these as true plugins where you can drop a new one in a plugin folder and the application picks it up, no recompiling no configuration, no restarting, just a black box that is self contained.

I would describe the second as a constraint rather than an abstraction. And I fully agree with this. The most obvious and probably most talked about benefit of FP would be avoiding state management complexities.

The third one is neat. But I would say we're already in danger territory here. Just writing a function and naming it properly is the minimal step required to make refactoring into a strategy or plugin pattern later fairly straightforward.

The followup is, if we have to change/replace said system, how difficult would that be? If scaling the simplest solution would be difficult/painful, then one can start look to higher complexity solutions. The idea isn't necessarily to always choose the simplest solution, but having it in mind can be helpful. It crystalizes one of the endpoints of the simple/complex spectrum and helps weigh the pros/cons of various approaches.

As a side note, it's sort of amusing that a lot of design focused interview questions are around things like "design a twitter/instagram like system that'll scale to billions of requests per day." I've never had to do anything like that IRL, but no interviewer has ever asked me to design, say, an invoicing system that gets called rarely. So perhaps one of the reasons the arc of software engineering bends towards complexity is that we're continuously rewarding a "build massive scalable systems" mindset?

I don't know that there is any readable wisdom that will teach you how not to overengineer or underengineer, such that with that knowledge, you automatically know how to achieve the right balance. It is likely a necessary part of the process to build out software projects that are in fact overengineered or underengineered and to intimately learn from that process as well as the aftermath how to tune one's own process to strike an artful balance between the two extremes.

put another way, the MS and Google teams you worked with were screwing up, but screwing up in a way that is necessary for people to learn, if they do in fact learn (they might not learn).

all of that said, starting out by intentionally underengineering, if you know that you are one to start overbuilding, might be a good strategy. but you might have to do some really huge refactorings subsequent to that when you learn your architecture is insufficient.

Enthusiasm and Energy needs to be controlled and directed by Experience and Good sense.

When everything is DRY and functions follow SRP, it's hard for your code not to be "future-proof."

Tests only when they make sense, not when you are mocking half the app.

Guessing the future and engineering to cope with it is a risky, error-prone business. Good engineering practice should always seek to minimise reliance on unreliable (prediction) data to create future-proof designs.

So I'd go with stopping as soon as it meets current use cases AND then shift gears to make it easy for someone else to pick-up in the future, e.g. write whatever tests are necessary and document thinking.

If it's easy to extend now, it will still be so in the future. Plus, there will be whatever learning has occurred since to make an even better job of it.

> How far should we go in making things generic?

Only if: 0) It's not only about guessing future needs. 1) The maintainers are over 90% certain it will make it easier for them to maintain and test now.

> Are there good resources for this?

There's a mountain of media on "Agile" software development, and it's different flavours. It's not particularly Software Engineering focused, but I enjoyed the "The Lean Startup" by Eric Ries.

Good luck and have fun.

It is more difficult to persuade others not to over-engineer. I have tried and failed many times to do so. In fact, if you try too hard you may just make them hate their job. Folks get into software development for a whole host of reasons and only one of them is shipping. They may not be satisfied with a job where they funnel requests from a PM directly into code with little creative input. I'm not sure I can give good advice on this front, other than to look out for certain red flags during hiring.

That's fairly true if you intend a monomaniacal focus on delivery and you are dealing with improvements that are truly irrelevant.

If other concerns like code hygiene have nonzero value, and/or if improvements have some potential relevance to delivery but are not unmistakably essential, then things get more interesting. Doing the least work possible to get by with a work item may be good for initial velocity but at long term cost.

It's probably easier in an environment where things like code hygiene expectations and review standards are well-known from a combination of express standards and team experience, but not all environments are like that.

Developers regularly hold religious wars on basic matters of code style. While you may profit from 'hygiene' there be dragons - developers often consider as hygienically critical such tasks as refactoring React class components into functional components with hooks. The industry is not full of Edsger Djikstras. In practice the sub-optimal greedy algorithm gets the job done since it encourages a straightforward approach to the problem at hand.

The secondary trap is that developers think they can amortize up-front development costs over a long period of time. Projects get the axe if they do not deliver. The longer you spend writing a feature, the longer you wait to test your hypothesis that it is valuable. In many cases the odds that the feature is valuable run below 50%. You may be laying a solid foundation when all that was asked of you in the first place was a shantytown.

Everyone thinks they are the reasonable person with discerning taste who only refactors code as necessary to optimize total cost. In practice that person is as mythical as a unicorn. It is very difficult to convince developers this is true, but true it remains.

If software engineering is engineering, they can. It may be a skill that is weaker and stronger between different engineers varying particularly by relevant experience, and it may require complex and context (including present development team) sensitive evaluations, but it is a real skill that exists.

> This is the evil twin of the notion that developers can judge what part of a system is slow without profiling.

Well, it's not, because there is no analog to the “without profiling” part. It's true that there is an analogous tendency of prejudging problem code, but certainly, at a minimum, hygiene issues can be discovered by experience of problems of development/maintenance stemming from, e.g., code duplication for shared functionality instead of use of shared abstractions.

> The secondary trap is that developers think they can amortize up-front development costs over a long period of time.

Note that when I said long term I don't necessarily mean a “long period of time” but “some period longer than the minimum development time of the present item”...

> Projects get the axe if they do not deliver.

Yes, they do, but plenty of real projects aren’t, especially at initiation, delivering actual value every iteration, just progressively refined demonstrations (this is particularly common on replacements that, for whatever reasons, can't go the strangler/ship of Theseus appproach, and on those projects the team often has an idea where things need to be for real delivery. It's quite possible for code that makes subsequent tasks more costly even though it saves time this iteration to delay real delivery.

> Everyone thinks they are the reasonable person with discerning taste who only refactors code as necessary to optimize total cost.

No, in my experience that's not even approximately true. Most developers I've encountered think that, in their current team environment, they individually have a natural tendency either to excessively favor direct solutions that produce ugly code with outsized downstream cost or to go a few steps too far with overengineering abstraction (and most of them also recognize that that that overall tendency is only on average, and that they also miss on the other side sometimes.)

My point is that when you get beyond the simplistic rejection of things which have no relevance to the task at hand (which isn’t a scenario that occurs all that much), evaluating what is the right balance is nontrivial.

Therein lies the question :)

> Well, it's not, because there is no analog to the “without profiling” part. It's true that there is an analogous tendency of prejudging problem code, but certainly, at a minimum, hygiene issues can be discovered by experience of problems of development/maintenance stemming from, e.g., code duplication for shared functionality instead of use of shared abstractions.

Even in this case we are limited. I've seen many cases where someone correctly identifies a piece of code that is painful to maintain and then dives in only to multiply the problem. We are even better off in performance land because we can test our code immediately after we write it to verify we haven't left things worse.

> No, in my experience that's not even approximately true. Most developers I've encountered think that, in their current team environment, they individually have a natural tendency either to excessively favor direct solutions that produce ugly code with outsized downstream cost or to go a few steps too far with overengineering abstraction (and most of them also recognize that that that overall tendency is only on average, and that they also miss on the other side sometimes.)

I don't want to wade into the thicket on this one except to note that it's possible for someone to articulate something they may even believe on some superficial level without truly internalizing that belief and that IMO I observe this behavior a lot.

I don't want to invalidate your point outright because I think it's valuable and I even agree that it's rare to encounter a code base that would not benefit at all from 'code hygiene.' All models are wrong but I think an aggressive focus on delivering functionality is a useful one in most contexts. Put another way, a good but imperfect rule of thumb is to refactor only in order to gain something tangible for end-users.

You aren’t gonna need it.

As with anything tho, ruthlessly cutting stuff or rounding corners is in tension with supporting future capabilities, sometimes you do need it it turns out.

I believe this is well documented today, but maybe not that much in academia.

What they need to avoid this trap is to talk with people who got burned earlier.

As a rule of thumb, don't design for future use case, ever.

Future-proofing is a fool errand.

I've also found that designing for unknowns can sometimes be resolved by asking questions about the unknowns until they're known. Sometimes, business stakeholders have an answer, they just weren't asked.

For example, in the 80's the linked list was my go-to data structure. It worked very well. But with the advent of cached memory, arrays were better. But my linked list abstractions were "leaky", meaning the fact that they were linked lists leaked into every use of them.

To upgrade to arrays meant changing a great deal of code. Now I encapsulate/abstract the interface to my data structures, so they can be redesigned without affecting the caller.

If you are doing a rewrite of an existing system or have many years of experience with a similar product then thinking ahead can save time. Otherwise you are probably better of not trying to be too clever.

The difficult part for most people is actually being honest in the process :)

Also Kent Becks timeless advice is good to keep in mind

1) make it work

2) make it right

3) make it fast

In that order. You might not need all three :)

And this is not an obligation. If you are really sure about the design of some component, you can also do it right the first time. But in most places, you usually don't know the design well enough.

Also, the imperfect solution can help as a validator. Have a problem that has a unique solution? Make a slow algorithm for it which you are sure is correct, then build a faster version and use the slow algo to compare for correctness. You can also use the slow algorithm to tell you about non-output values that both algorithms compute implicitly or explicitly.

I usually try to support the opposite idea by bringing agile to the table. Agile doesn't say "please don't overengineer" but "this can change a lot in a couple sprints, requirements may be different, there are more (or none anymore) use cases now" and hence why to spend so many resources doing something that not only doesn't cover all potential unknown use cases but also adds overhead when reading the code for no benefit. For some teams this has clicked very well. Appropriate agile training can help a lot.

However it doesn't always work and if the person(s) doing this have higher ups backing it up (even if unintentionally) you're fighting the wrong battle. If you can't change it, move teams or to a company where money is a bit tighter and/or where these behaviours aren't tolerated - in other words, where things needs to get done and there's no time for experimenting with things that most likely won't yield returns.

Then understand a new problem.

Repeat.

If you then start doing things like externalizing inputs, writing tests then when your past solutions become future problems.. You will have created the guide rails on how to solve both the past problem and the future problem together. This is the futureproofing that you should be aiming for. Make sure you write at least a few unit tests that mock out the important logic parts, and write up a short 1 page document about the intent of the program, maybe with a picture and drop that in the README.

* Focus on the current assignment. Implement it using clean code principles, don't overthink the problem.

* Rather than spending time on design decisions, allocate time on handling edge-cases. These will save you from PageDuty alerts.

* Plan excessive for future use-cases only around data models that insert/read into the database. Data migration is super-painful. A more generic design around your database is almost always preferable.

* Have a feature toggling[1] service in your codebase. It will provide you with a better understanding of how you can implement new features alongside existing codebase in the same branch. Releasing features from long-running separate branches is almost always a wrong decision.

* Always keep in mind time-constraints and how urgent is the requested functionality. Don't let perfection be the enemy of productivity.

* Have a process in place that allows for the technical debt tasks to be tackled independently. It helps fix some bad design decisions, which become apparent in light of new requirements.

[1] https://martinfowler.com/articles/feature-toggles.html

Use TDD and only write only just enough code for the currently known required functionality. When you get to know more required functionality the tests protect you from breaking existing functionality and you can extend your code to support the new use cases as well. At this point you should make your code just generic enough to support all known use cases without code duplication or too much boilerplate code. If you can support functionality in more than one way you can decide what way to choose based on what you expect in the future. But choosing the most simple solution trumps attempting to future proof your code. It turns out that predicting the future is quite hard and there will be new feature requests that nobody had foreseen and code that has been made as generic as possible will not handle this well.

Spiderman says that with great power comes great responsibility. The converse also holds true. With great responsibility comes great power. You cannot just pick the easy part of TDD, be irresponsible and expect to have any power. The less easy parts of writing a test for every use case and of refactoring all the time make the practice of not attempting to guess the future possible. If you leave out the prerequisites the end result will not be so very pleasant.

It implies:

- Behaviour can and should be compartmentalised

- Certain types of efficiency are ignorable

- Data structures are better off being relatively simple

- Behaviour is able to be understood before the system emerges

- The system doesn't fundamentally need a lot of mutable state

- Dispersing functionality across small, atomic functions (that obscure sequential flow and state mutation) is good for the code

- It is easy to extract the functionality of this system into pure functions

- A high-level veiw of the system is unnecessary (!)

- In this system, most bugs will come from small units, not interactions between units

- Behaviour of units is likely to be relatively unchanging

- Refactoring primarily happens between interfaces, not to them

- Test rigging is cheap and easy at every level of abstraction, or at least that it's better to contort your system into a structure where that's true

None of these things are a given.

I kind of feel you live in some kind of alternate universe. None of these sound true to me, at all. The most important misunderstanding that seems to be going on here is the assumption that in TDD it is a given that one is testing single classes and/or methods. This is not the case. In fact, in most cases it is much more beneficial to test a set of classes/methods at the same time in a way that is representative of something that the customer values.

Honestly, I have seen TDD work so well in so many different circumstances that I have started to consider people who do not do not write test for most possible scenarios as being on the not-so-very professional side of things.

The first is: Requirements change.

For many reasons. User needs change. Business processes change. Technology evolves.

If you're going to keep up, you must design code that is easy to change.

Code that is easy to change tends to be code that can work with lots of different code.

Code that can work with lots of different code tends to lean towards the general end of the spectrum, rather than being too specific.

Designing code that is modular does not mean that you need to over-engineer, and it doesn't even mean that it needs to be used more than once.

It should mean that the code has locality: The ability to understand the full effect of the code without also understanding the full context of the code around it or the full history and future life of every external variable it uses.

It may sound to new developers like I'm talking about something complicated, but it's the opposite:

Code that can be easily adapted to future use-cases tends to be more simple. It tends to know the least about it's environment. It tends to do only one thing, but do it so well as to be perhaps the optimal solution.

What follows from the first principle is perhaps the most important principle in software development. Remember this and you'll find yourself needing to do a small fraction of the work you once did to produce the same value:

A small change in requirements should lead to only a small change in implementation.

I was talking to a new guy that just joined my team yesterday on this subject. You really cannot predict the future.

You could also look at it from one other angle. If you are only building the bare minimum to satisfy the requirements, that is a lot less code you are writing. If you need to replace the system, that is a lot less work to go back and rework.

I dont like "used the simplest solution possible" advice, because people who I claimed doing that in real life tended to do unmaintainable spaghetti mess. It was "simple" in the sense of not having abstractions or nested function calls, but hard to read and understand big picture. Sometimes generic is a way to untangle such previous spaghetti mess. As in, it is second step on the road that requires 2 steps till it is really good.

Understand politics. A lot of those "future cases" are things that analysis or management requires initially or indirectly. They are also often result of trying to hit vague requirements - you dont know what customer really needs in enough detail, so you do it configurable in the hope of hitting the right place too. They are also situations in which people burned out in the past or special cases of special cases.

Otherwise said, the complicated thing is often requirement, initially. Someone had reason to ask for it or thought to have reason. It gets forgotten and ignored after a while in which case it is ok to cut it off.

Invest in very good testing, especially higher level integration / system testing.

Invest in a good dev/staging setup for your production environment, and also try to make rollouts and rollbacks automated and as painless as possible.

There will always be the need to change stuff, so get the pieces in place to make changes easier code, easier to test, easier to deploy, and easier to back the fuck up when you inevitably cause something to burn.

Of course the issue you run into is that the problem software is trying to solve changes over time and then the original abstraction which were correct become wrong over time. Then you should advocate refactoring to the new abstractions if possible.

Immediately. Never design for a future use case until it's a present use case and you're implementing it right then.

> How far should we go in making things generic ?

It depends on what it is. By not designing a thing to be generic up front, you have to figure out what n=2 looks like. Is that a function? A class? Copy a little bit of code? Then n=3. Once n=10, I feel like I have a good idea of the problem and how to make it generic, and it's rarely what I would have thought at the beginning.

Sometimes n never reaches 2. Then you've saved a lot of time. Also, you realize when you have to change things - maybe it's once a release, or very frequently. Things that are touched frequently probably need a refactor.

My rule of thumb is: never make tomorrow's possible problem today's complexity. If you design for future use cases, not only will it take you longer, but your code will inevitably be more complex than it needs to be, and therefore have more bugs at the very least due to that complexity.

Problems arise when people reach for the Cool Tools and start building Webscale things that can handle 100M operations every second. ...but the customer only needs the system to handle 4 users who type in everything crap by hand. But hey, it has a clustered database and Kafka and Kubernetes and looks REALLY good on your Resumé.

When the scale is determined, I personally like the mantra of "Make it work first, then make it pretty". First build an MVP (or an ugly Viable Product), that proves your plan actually works, then you can iterate over it to make the implementation cleaner or faster or have better UX.

If you get stuck making everything super-generic and able to handle all possible cases, you'll spend time bikeshedding and never get anything deployed. Just Repeat Yourself with wild abandon and copy/paste stuff all over until your Gizmo actually works. You can spend time figuring out which parts are actually possible to make generic later.

For a long time we would "pretty it up at the end", in one of my coworkers words, which lead to a ton of horrible UX decisions early on that required major legwork later. We switched to doing ux-driven development from the start and it's saved us a ton of time and saved ourselves from poor decisions haha.

It depends on the customer and project which parts I focus on making "pretty". If it's a data-intensive thing, I might spend time optimising the protocol, compression and making the data pipeline robust. For a web app I'll spend more time making it more usable by streamlining the most relevant flows (which I know of since the client has been testing the (M)VP already).

Ask yourself the following questions: * Are you an experienced developer on this particular problem domain? * Do you have good understanding of the future use cases, and the data structure when the new feature is added? * Will the new feature make business or technological sense?

Unless you have a concrete “yes” on all questions above, your design should go minimalistic. Without good understand of the new feature, the code base will likely need refactoring when the feature is implemented. Your effort of going the extra mile (which definitely should be applauded!) is better spent on making the code easier to read and refactor - better tests, better documentation, code review to propagate knowledge and catch bugs.

Another thing, extensibility is sometimes in conflict with readability! Here’s an hilarious example: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...

In my experience following categories of people tend to fall into the trap easily (and it is really hard to convince them):

- Corporate people.

- Business school people.

The following can amplify it:

- Fund raising... you were successful in raising money so that means there is a market and people love your product ... well guess what? you are wrong.

If you are in such a situation, debating is not useful. Changing this mindset is only possible when you hit the wall at least once in your life. BUT ... you can accelerate the process of realizing it (before it is too late). Try to find a way to make people realize that all those bells and whistles and designs were not necessary cause nobody actually asked for and we are changing it anyway now.

* The nature of your "system" defines how much attention you should pay towards futureproofing. If it is an end-user app/feature only do what is required and nothing more. You are solving one instance of a (maybe) small class of problems. The future will dictate what needs to be added/modified/refactored when you reach that point and not before i.e. no predictions unless you know the domain well.

* If you are building an "architecture" component like OS/Framework/Library etc. then you need to pay attention to generality and extensibility and design with futureproofing in mind. Use standard best practices like data/api versioning, narrow module interfaces, information hiding etc.

* Always focus first on Readability and Maintainability to the exclusion of everything else. Only when you hit a wall with respect to aspects like Performance etc. do you go back and redo/refactor the code for the newly prioritized requirement.

Modify the code when the use cases change. In most of the times, open-closed principle is a trap.

However, over-engineering is not only a technical problem. It's more of a problem on project / people / finance, here are some examples but it would definitely not limited to these:

1. The margin is big and the team is big, so we need to keep these people busy.

2. The current stakeholder will cover the budget these iterations until we finish these features. after that, the maintenance costs maybe partially falls on us, or there's will be no budget at all. The less problems in the future, the better.

3. The current budget is for functionality A and someone pays it. And we plan to implement another feature B that is similar to these, and the budget comes from our own. Better make the solution generic so we can use it almost for free.

4. The list goes on...

Better fix those root problems first.

* A demo is a thing you can show. The first demo is usually

   - the program will compile, run and print "hello world"

* Demos happen frequently. For a developer each day, for a team each week, etc.

Why does this help? A demo, actually showing something, is the only enforceable token that commits both part to the contract. In that way it is almost a currency. No stakeholder, whether manager, sales or developer can argue - either the demo was met, or not or was not clear.

This is much like sprints, test-driven, but a demo has the contract aspect.

Demo-driven reduces many of the causes/motives for over-engineering. * Is the over-engineering bit in the contract? Why not? * New function requires a new contract so no more last minute changes. (Been there done that). * Stakeholders gain experience designing demos. * Demos are adaptive. They provide tactile feedback.

My 2 cents

With modern higher-level languages and scalable cheap hardware, this motivation should have gone away and we should be writing code that is relatively easy to re-factor.

If I don't know that we will need this thing in the future, it doesn't go in. Simple. If 6 months down the line, we now need to add the new feature, I like to think that my code is largely maintainable enough to refactor it to add the feature.

The only exception I can think of is where something is designed to be extendible like e.g. Instagram filters where you might have 10 when you launch but you know that you will have more in the future so you write your code to allow additional filters to be plugged in relatively easily.

Things get over-engineered because too much time is allowed for the task.

The forcing function that's helped me is simply giving me and my teams small but meaningful time boxes to get a feature done. Sort of like "sprints", but with more teeth. E.g. An important feature is going to get announced in a newsletter ever 2 weeks. Sprints too often don't seem to focus on the shipping to customers part.

So we focus on shipping the smallest thing that could possibly work in an arbitrary time box. You know users need X. So you make X or X' or X'' - some version or whittled down version of X that'll relieve user pain. The time box does works wonders from keeping over complexifying from getting out of control.

Truth is most projects will be shelved before even they are complete.

You'll be taken off some projects because of financial or political BS.

So start with the assumption, there is limited time to deliver - now ask yourself - how can I maximize the effectiveness of that time? How can I do what truly matters without going into the rabbit hole of optimizations and using the best thing possible everywhere.

And once you've done that, you can always come back and improve the thing if the project is still around but most likely it will not be. Either you'd have switched the company or company would have switched you or the project has switched the company.

Architectural boundaries are hard to change later. Drawing the dependency graph and isolating nodes that can change independently is where the majority of our effort should go (imo). Even still, simple is less risky than complicated. Anytime there are more moving pieces than necessary, there is a risk of an unexpected requirement blowing a hole in a design. So identifying those dangling pieces and spending a lot of thought and energy in removing them is where I've found it to be rewarding to "overengineer".

In practice goals start to move after horizon/2. And a new document needs to be created to capture where the business is doubling down and where it is trimming.

Ex. if you have a problem to solve that goes like: Must create store with products that are blue only.

Then you'd break it up like:

Create store Filter products (Blue only) Create products in store

Then when you start coding you solve nothing more than what you put down as each task.

Ex. you don't do anything more than what's required. You might think you need to create a store and stores are businesses so you need to create some business wrapper etc. but nope. You don't need that until a client comes one day and requests it. Right now all you need is a store with blue products and that is all you're going to solve.

Often when you over-engineer something then you never need the extra abstractions you created.

You need to always ask yourself "What are the reasonable future use cases? How much will it cost to add infrastructure now to enable each of them, and how much will completing the feature in the future cost? How much will it cost to refactor my code later to add it if I don't make preparations for it now?" and only prepare now for things where not preparing would cost a lot more.

I'm finding the game of Go (Weiqi, Baduk) to be a great way to train in this skill, because it's all about seeing potential moves and deciding not to play them yet, and judging if a move should be played sooner or later and how much of a shape should be built now to enable it to be built later without wasting resources on building all of it now.

In my experience, keeping behavior static/hardcoded is the architectural equivalent of avoiding premature optimization.

[0]: https://max.engineer/cms-trap

Thus, we have every company it seems these days running around trying to implement microservices because “our architecture is like Netflix’s”. No, that LOB CRUD application isn’t like Netflix and you don’t need polyglot storage, Kubernetes, and microservices to capture input from a web form. However, one of the managers between the architect and the CEO (see: Peter Principle) is extremely impressed by this idea and can use it to advance their personal career.

My typical approach is to log all the commands to the file(s), and keep the system state in memory. When the server restarts, it simply replay most of the commands (skipping those intended to cause side effect)

This is like simplified event sourcing, or command sourcing.

I'm not relaying on any framework to build the backend, simple library handing rest and websocket API is enough.

I'll open source it soon but it's mostly as simple as it sounds, with some optimization to reduce disk space consumption after they becomes a problem in practice.

[0] https://en.m.wikipedia.org/wiki/Test-driven_development

Here's one Kevlin Henney's lecture [1] that crystallizes it. It took me a long time to find it, so you are welcome.

Once you start naming things like this, adding future use cases becomes far less risky and thus you don't need to waste time on them.

https://www.youtube.com/watch?v=tWn8RA_DEic

A larger codebase than solves the same problem as a shorter codebase is almost always worse, has more bugs, and is harder to maintain. Code is the enemy: the only good line of code is the line that doesn't exist.

Once you have a established a pattern of how the system is regularly extended, then you can use that to make predictions about the future. Keeping your codebase well-tested, small, and light will do far more to help you respond to change than guessing.

How do people “defend” that in big Corp?

Boat the technical roadmap and requirements with fantasy future features, so they can say “well that framework doesn’t support X, and our framework will support X”. It doesn’t matter whether X is useful in practice or not, since the managers don’t always have the power to make that call.

2: It's easier to refactor code with good automated tests, like unit tests, because you can push a button and know that it works.

3: Make sure that your startup order is well-defined. It's easier to debug a well-defined startup order than a startup order where everything is implicit.

4: Know the difference between a design pattern and a framework. Frameworks don't replace knowing how to use a design pattern correctly.

One thing too keep in mind is that the tests should also be kept relatively simple. I have seen codebases where there were so many and often complex tests that modifying the actual code was easy but changing the test would have been prohibitively expensive.

Any insights into how to do exactly this? Any rules of thumb or guidelines? Also what is not considered complex to some adds a cognitive burden to others.

I think the sense is mostly developed not by successful designs but by mistakes and the subsequent refactorings.

The only “simple” advice I have is that in FP the simple and decoupled seems to happen without added work, while in OO it’s quite a cognitive overhead to avoid tangling things up. So my only insight is perhaps “make everything simple functions and shy away from state whenever possible”.

Don't do work before you need it, but also don't give up optionality unless you're getting commensurate benefits from doing so.

But there is a certain joy that comes when people ask “but what about XYZ” and you can respond “Yup, we thought of that!”

Granted, I work mostly on developer facing tools and services, which makes it much easier to anticipate the needs of my customer since I too am a developer.

And even with that caveat it’s not always a slam dunk... but it certainly is possible to anticipate requirements in a useful way.

The more time you spend thinking about the business and how what you're building will support it the better judgement you'll develop. You might not be able to articulate or argue it but you'll have an instinct on when an abstraction is going to be useful vs brittle.

If you build systems with these principles in mind, you can create systems that are extensible, without creating technical debt and YAGNIs.

It’s actually not that hard to refactor things when you do need a new feature... and who doesn’t love refactoring? It’s fun!

It’s also easier to refactor when your code is smaller and simpler because it’s only been coded to do the things you actually wanted it to do now.

If new requirements come in it's anyway lots of typing. You might as well spend the effort only when the full situation is known.

Accept the abstractions if: - the stack trace depth is never greater than double - the total code size is decreased - performance/memory hit is less than 10%

Thank you for bringing this up.

[0] https://twitter.com/KentBeck/status/250733358307500032

It is really hard to know what other people want or what they mean. If you can really understand what someone wants, perhaps you can avoid write 50% of a system you initially imagined.

Changing in the future costs something.

Designing for future changes up front makes sense if cost(future change) > cost(future proofing)

SAAS? Do virtually no future proofing.

IOT, do some.

Space probe? Do lots.

Also, if you haven't built quite a few relatively similar systems, don't do future proofing without talking to people who have.

They're as good as you make them. Abstractions aren't bad by themselves.

> Copying code for different business purposes is good.

Only when it makes sense, ie when you can't make a good abstraction.

Abstractions end up wiring things together that blur business rules.

2 - Write the simplest code to fulfill this specification;

3 - Improve on what you wrote so it will express better exactly the specification you have so far.

Other thought: good engineers can guess future cases more accurately, that's why they're good engineers.

A well-maintained, well-pruned test suite.

Writing tests forces you to clarify -- to yourself and to others -- precisely what this piece of software is and is not going to handle.

When I joined the attribute system had been build with a beautiful UI, and the backend was mostly working for managing attributes, but that was pretty much it. The first feature I was working on was showing products on the store, and for this the idea of attributes made sense. If you are selling a product in the "OLED TV" category you probably want a "Screen Size" attribute, and to be able to use it to compare against different products in that category. Through the platform we had maybe 500 product level attributes, with more being added all the time, so having them hard-coded wouldn't have been manageable. That was pretty much as well as it worked though.

Sellers needed to be able to manage their stock through the system, so on the warehouse entity there were attributes describing the number of products in stock, lead time, how often they restock, etc. The attributes didn't really have validations, but they had types which described what UI element should be displayed when they are entered. However all of the validations around that were at the whim of the front-end, and in some cases it would send what you would think should be a numerical type as a string (and then if you try to change it something would break as that expected it to be a string), so doing any kind of calculations or logic on the attributes was basically impossible. In the end I just added db-level fields to the stock entities, with validations in the backend to make sure these were as expected. The backend was a Rails app, so this took 10 minutes vs days trying to coerce the attribute system into doing what I needed.

As it was a Rails app we couldn't actually name the model of these Attribute, so had to give it another name, and whenever someone new joined (it was an early stage startup, so had high turnover) we had a 30 minute discussion explaining this. I never got the explanation of how the product owner expected logic to be attached to these attributes without a developer doing any work, but I'm sure they had an 'ingenious plan' for that too.

Needless to say, the startup burned through all it's funding without even launching, then managed to convince the investors to give them a little bit more, launched a half working product, and it turned out nobody wanted it.

Never make things generic apriori.

requirements -> tests -> specific code

if i reach the same code more than once, refactor and bother to generalize....

How do you go about planning your tests / separation of concerns. As in do you only write tests for your service layer? I find I'd be wasting time to write it at the controller level/route level.

What about the DML schema?

Personally I always start at the database layer and work up because then I know at least what data and models I'll be working with

Often people refresh the browser or re-run their CLI program until their feature is finished.

But if you think about it, every "refresh to check if it works" is just a manual test. TDD is just making that manual test automated.

1. Write that test (that you'd anyway have to run manually)

2. Write code until test pass

3. Repeat 1 until done.

Code that aren't designed with a test-first mentality is often really hard to test and require complicated tools or need to mock the whole world.

For the examples you've mentioned:

- I'd unit tests the db service layer (I.e. functions to fetch from db, make sure schema is valid)

- I'd unit tests the various API queries (I.e. filtering, pagination, auth)

- At the controller level, I'd just unit test the business logic and data fetching part.

- Then I'd add a few E2E tests for the UI and user interactivity.

But if you think about it, any of these tests would have had to be run manually anyway. I.e. You'd probably have queried your API with various options and refreshed the page(s) a few time to make sure data was fetched correctly.

If your test is more than 5-10 lines long, you are probably trying to do useless things like mock half the app and you should have a method that returns simple data / services that don't take a 1000 other services as dependencies.

TDD is only a good choice when you have a clearly defined problem and know how you solve it.