I found myself nodding vigorously to Ms. Atkinson's (fictional) responses.
> You fixate on some technical solution and then completely forget what is the context in which you're building it. Good software designers basically treat all problems they face as ill-defined, even if they are fairly specific.
Oh my god, I nearly fell out of my chair. YES.
What we need are tools for software development that are highly flexible, easy to use, and safe. We need to put those tools in the hands of someone like a software designer from this blog post: A domain expert (and/or intuitive listener) who deeply understands the business, understands their customers' needs, and can hook up whatever logic is required themselves.
(And uhh, shameless plug, I'm actually working on a product right now that attempts to realize this dream. We're hiring platform engineers. See my profile for contact info/links.)
Can you please explain what differentiates your product from previous attempts to realize the dream? E.g. 4GL, Visual Programming Languages, business rules engines, Access, SharePoint, low-code or no-code platforms, etc.
So, in some ways, it is not that different. It uses ideas that have been around for years: The core of the product is a visual programming language, plus a database, plus a low-code Python editor. All of which have, obviously, been done before.
But the problem is, no one has put them all together in quite the right way to satisfy our needs.
- We needed a platform that would let our non-programmer staff create complex integration-related features for customers. We needed an accessible VPL that was easy to get started in, but powerful if power was needed.
- We needed a lot of integrations to not-particularly-popular business software (which means, we needed the ability to write and/or extend them ourselves); we needed the integration editor to be accessible and easy to use by junior programmers.
- We needed data pulled from integrations to be stored and kept around for a while; we need to look up individuals, see their history and a history of triggered VPL logic; we needed to be able to update and bulk-execute VPL scripts against the data store; we needed full text search capability as well.
- We needed workflows/data organized into tenants with a user auth scheme that would let us invite business customers to see just their data and no-one else's, with view-only permission as well.
- We needed something that could scale up to handle a lot of work, and still be at a competitive price.
There isn't a platform that has all of these things. One or two, sure. But not all of them. My hope is that in 5 years b2b saas companies won't have to build much integration software themselves; there will be us, or someone like us, that'll handle it for them, and they can focus on whatever it is they actually made the company to do.
I find it odd that this doesn't make mention of civil engineering or construction engineering in the architecture analogy. Like, absolutely the artificial world should be designed with objectives like appropriateness, but that doesn't mean you don't need to have people whose roles focus on the technical approach to carrying out a design.
We _have_ product design experts and processes for software products, but that's not a reason to not have engineers. And as to why they're not celebrities typically ... I think that's also in part because the companies that produce software want to control their brands. This isn't a threat for architects; BMW doesn't lose anything by us knowing that Zaha Hadid designed a building for them. If she had then designed a building Toyota, that wouldn't have been a problem. Their value proposition isn't about the building they work from. But if you knew and cared about the name of the specific designer behind your favorite software product, and they left the company ... that would be a problem for that company. Part of the prestige associated with their core offering is able to walk away.
> I still feel like this way of thinking can work only up to a certain point. Don't you need to get to a more precise problem definition at a lower, more technical level?
> Not really, no. When you're solving a problem, even as you get to a more technical level, you always keep in mind why you are solving it.
I have mixed feelings about this. On the one hand this pretty much describes many applications built with a lisp. Everything is built to solve the original problem. Every construct made ad-hoc only handling the cases that are relevant, even making up effect partial languages suitable for the subject at hand.
On the other hand, we don't end up with globally shared libraries that have 'completely' solved a particular technical problem space. We actually lose the 'science' of computer science and everyone is an application developer.
Based on the popularity and effectiveness of libraries and ecosystems, I'd say our universe is winning. I do however enjoy quickly making partial implementations that perfectly satisfy an application's needs up and running quickly. Sometimes the complete solution would be unapproachable but the needed one clean and simple.
Perl is an exception--it has a vast ecosystem of partial implementations.
While that's a good point, I think that many libraries still have the overall design goal baked into them. These aren't independent libraries that solve an abstract problem somewhere out in the aether, but rather libraries that are intended to fit into specific designs.
Consider a serialization library, for reading and writing objects from memory. This library could aim to have a minimal memory footprint, and may fit into the design of an embedded device. This library could aim to have maximal throughput, and would best fit into the design of a server application. This library might aim for usability by new users, and handle its own memory allocation for the serialized object. But, that would imply serializing the entire object at once, rather than streaming it to disk or to a network socket, so that impacts what designs can use it.
Libraries tend to be implemented such that they fit neatly into some original design. Figure out what that original design is, and that gives you a better sense for how it will fit into the design you're making.
> On the other hand, we don't end up with globally shared libraries that have 'completely' solved a particular technical problem space.
IME the more general a library tries to be usually the worse it is to actually use. Opinionated AKA designed libraries that have some specific intention behind them are more immediately usable and it's much more straightforward to see the pitfalls in using them as they tend to typically be intentional.
It can go the other way, like I started of creating a POC of a 3D app with three.js and then simplified the code by just creating my own typed array that is send to the gpu.
There are many software designers, they are just not famous because no one would understand what they do.
Huh. I don't see much of a contradiction here. Even the most design-y of software designers isn't going to be picking out artisanally selected sand grains for inventing a new kind of silicon-based processor. At some point you say, "Well, these generic components are good enough, let's use 'em."
That choice of components still requires keeping the purpose and context in mind, of course. But if I want to get anything done for the user, at some point I hit bottom and return my attention to getting something done and out there.
Not a single mention of data structures. It really is the heart of software when done properly. You can often tell if someone is a subscriber of that view depending on where they place their data structures in the source files.
Once the data layout is properly designed, the data transformations required become clear and the rest of grunt work.
> Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.
~ Fred Brooks
> I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships.
> "Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming." ~ Rob Pike
Agreed 100%. What are the common cases we need the data structures to optimize? How does this penalize other parts of the system? How do we make the data structures extensible for new features in the future? Does this data structure prevent us from making any other use-cases work well? How complex is the resulting system going to be, and will this cause us any future maintenance issues? Does everyone on the team understand the approach we are taking?
My experience is that good teams start by asking lots of questions. The design may not be entirely formal, but good developers do go through the process of figuring out their internal list of requirements needed to satisfy the requirements of the system as a whole, and try to make sure that the decisions made are well thought out and don't box the codebase into a problem space in the future.
All programming is made up of data structures and algorithms. A program could be all data structure, a mixture of the two, or all algorithms. In a perfect world, where a team has infinite time and money and a perfect understanding of the problem, it would be all data structure and no algorithm.
This may sound good to someone but it doesn't make good sense. Data structures aren't free, they take space. Therefore there is a trade-off between space and compute complexity. And ultimately data structures don't transform themselves, an algorithm has to do the transformation...
Once you modeled your problem into some data structures you now need to make the decision of whether or not the transformation between input and output is easily achievable with an algorithm, if not then you need to further break down the problem into more data structures or your design is flawed.
Yeah, I agree with most of what you put here. I'm mostly speaking to the articles point where we are in an alternate reality with different built up structures. When thinking about going purely data structure, I was thinking something like Church Encoding[0] or Prolog.
Is that practical today? No. The effort required to get a full program created at that level of detail would be enormous. Technical debt is in itself not a bad thing, and I'm not saying anyone in this realities goal should be no algorithms.
Well no one has infinite time and money and a perfect understanding of any complex problem yet, but I think the F# DDD[0] overview can show an example much closer to more data structures and less algorithms.
I don't see how the other universe ever designed CPUs, RAM, or desktop computers without the specific end goal in mind. Perhaps they're drowning in purpose-built hardware?
The conceit isn’t that there are no engineers or engineering disciplines just the focus for software development as a discipline was design rather than engineering.
I believe GP understands that point and is arguing that without the specific philosophies that led to "software engineer" titles, you'd be hard-pressed to build the general purpose computational machines that birthed our modern software industries.
I still don’t follow that. If engineering still exists the philosophies still exist and the interview mentions what could be interpreted as the forking moment in the 1960s. By which time I doubt a subtle emphasis shift like the one suggested here would prevent general purpose computing from continuing.
Where's the line? Is my point. How do you build an operating system when you're starting from "What specific task does my end user want to complete for this bit of software?" You never build general capabilities; only specific point solutions.
In other words: this article was written on a computer built in our universe. In their universe they'd be stuck on electric typewriters.
Even an OS isn’t a general solution. Take ChromeOS versus the OS underlying a PS5.
Also what specific things does an OS let you do. Think about the end user from that perspective. What do they want from an OS? It’s hopefully quickly apparent that you can consider these things in the way the article suggests. It’s just the emphasis is slightly different.
I certainly think that emphasis difference would result in different OS designs but by no means would you be stuck with electric typewriters. That feels like a very uncharitable reading.
I think you're being charitable enough to read this universe's methods into that one, which aren't compatible (and this incompatibility provides the contrast).
For example:
I still feel like this way of thinking can work only up to a certain point. Don't you need to get to a more precise problem definition at a lower, more technical level?
Not really, no. When you're solving a problem, even as you get to a more technical level, you always keep in mind why you are solving it. I noticed that software engineers in your universe often ignore this. You fixate on some technical solution and then completely forget what is the context in which you're building it.
As you can see, this isn't the sort of mentality that gets you a general purpose tool. You might get JPEG, but you won't get ZIP.
Makes a lot of sense to me, most of it. I think they idea that you can't learn important software engineering/designing lessons from a book is overstated though.
But it is true that most of the existing books probably don't include some of the types of case studies and lessons that you would get from a job. But I think that textbooks should simulate those things with exercises. Of course a textbook can't include real dynamics or interactions but it's not true that it's impossible for some of those lessons to be in a book or at least touched on.
I mean, a textbook could actually emphasize things like digging in and iterating to really understand the problem in depth. The criticality and difficulty of understanding the problem and finding the right framing for the problem, realizing that is something that should be deliberately shaped to enable the solution.
To me the most challenging thing has always been the power imbalance between myself and the managers or users who are incorrectly framing the problems, and the struggle to negotiate an adequate problem definition that will enable a good outcome.
I recently did an interview on this topic [1] and I was surprised to hear the designer say, "If the developer tells me to design whatever and they'll build it, that is a huge fail". I kinda assumed designers were in their own world where we (developers) met at the DMZ, but his point is we are all part of the same team. The more I thought about it, the more sense it made.
That hardline hand-off from design to developer is only desired by arrogance (arrogant designers or arrogant developers). It should be obvious on the face of it that good design is about compromises between perhaps pure goals and what's attainable with the tech you have AND what's attainable with the product team you have.
Ditto for hardline requirements from project management. It's either a CYOA strategy or a holier-than-thou mentality.
Well said. Funny we realized long ago that you can't just have a backend team build an API then throw that the wall to a frontend team to use it, yet somehow, we often try to do that with design. To achieve what you describe takes more than a conversation, but a healthy and active relationship.
Some of the moments I enjoy the most are where I sit in a room with product and design for an hour and we riff on a design. The feedback loops between disciplines are where the magic happens and talking through things in person usually results in a lot of improvement fast.
Exactly! Faster cycles means less waste and more productivity, not to mention better solutions and happier team members. It takes work for sure to build those relationships, but everyone is better off for them.
Often overlooked, the core tenet of engineering is public safety. This is why professional engineers are required to obtain a license, register in their practicing jurisdiction, and are held liable if their work fails to achieve a prescribed consensus of acceptability.
Imagine if software was held to the same level of scrutiny..? Software is carpentered, some is designed, and little is actually engineered.
Engineers don't fail forward, early, fast, or often. They don't move fast and break things. Not engineering software is great, because if you had to pull permits to build your software, the web would still look like 1996 and my phone would have a keypad.
I have always treated my professional work with software as very serious business.
We aren't developing anything with direct life-safety consequences, but we are dealing with PII and other people's money. I see no reason to relax due diligence on anywhere along this spectrum. It's either correct or its not, and the consequences for getting something wrong are universally bad for all involved.
Now, I also strongly believe in the idea that software development is inherently an art form. The nuance is that within this art form, there are still very rigid conceptual frameworks you need to work with. For instance, designing the UI for any application is more-or-less a pure art form. Developing the SQL schema backing that UI is arguably something that is 100% deterministic and can have mathematical proofs established which prove its correctness one way or another.
The key to the success of any software project is blending these two worlds together in a sustainable and responsible way.
> Developing the SQL schema backing that UI is arguably something that is 100% deterministic and can have mathematical proofs established which prove its correctness one way or another.
Maybe I'm misinterpreting your point here, but the way you create the different models, compose the modules which will use them and represent the abstract entities and processes that execute when that UI is interacted with can be done in many ways, both well and very badly, depending on the measurements you take to evaluate it.
All the different terms to consider 'clean', 'maintainable', 'readable', etc. code, I consider to also be somewhat abstract to define in an art-type of way.
The reason I mention this is that the 'visual design' of the UI is not so much a software work as much as a UI/UX designer work. Bringing that design to life in an application using actual code and not a design tool is where the same principles I mentioned above come into play, even if the actual 'visual design' was completed by a different person which might not code, or the same dev.
I would argue that there is a very specific form of normalization that any problem domain should be modeled in - 3NF or BCNF. This is a universal idea because of how it allows for relational algebra to easily produce any higher-order functions over the dataset. Poorly-normalized data is where all of the code & SQL spaghetti originates. These forms of normalization can be specifically tested for and validated.
If you get a super clean domain model in 3NF, you can be assured that writing business logic on top of this will be a breeze. Getting relationships between domain types incorrect, or having facts on the wrong types, is how you get a vicious cycle complexity layering on top of itself as you try to compensate for bad foundations.
One thing that burns a lot of developers is the need for circular dependencies. You cannot say that circular dependencies are disallowed when trying to model the real world. An example in banking would be Accounts can have many Customers related to them, and Customers can have many Accounts related to them. Attempting to "choose sides" on this is folly. The most correct path is to create 3 collections - Customers, Accounts, and AccountCustomers (or CustomerAccounts). Getting this one thing correct can save you years of developer hours on the bigger projects.
If you are using something like .NET, you can abuse LINQ to produce any arbitrary projection you desire in a few lines of code. If you want your business people to have a hand, you can expose SQL as a scripting language against your properly-modeled domain. There are so many good reasons to slow down and pay attention to what you are calling things (and what boxes you put the things in).
To put it more briefly - The argument I am essentially trying to make above is that there is actually one truly correct way to model a specific problem domain and you can test for many degrees of that correctness. I will admit it is never absolute, but I feel it gets damn close if you are thorough in applying single responsibility principle to the logical business facts.
> Developing the SQL schema backing that UI is arguably something that is 100% deterministic and can have mathematical proofs established which prove its correctness one way or another.
Maybe you can prove what software does, but "proving" that what it does is the right thing is necessarily subjective and grounded in reality; mathematical proofs need to get dirty.
This is my pet peeve with many software developers. They develope software like a photo lab technician develops a roll of film: Let's see what result we get now, if it is problematic move to the next thing.
I think bot good engineering and good desin practise is the opposite of developing a picture. It is really iterative process, trying to factor in as many of the important aspects as possible and try to get the one solution where all of these aspects don't fight each other, but strengthen each other.
Many software developers I met are just focused at what framework/language/design pattern/etc they are going to throw at the next problem. Of course that is all exiting, but what if the thing you are tackeling could be solved so much better, if you just killed your darlings and tried to find the best solution?
Sure you cannot learn $framework you were curious about on the way, but finding elegant and good solutions is itself a skill that needs training.
The projects I have been involved with tend to have high level design discussions at the beginning. Working in a team has that as a necessity, otherwise we can't figure out how to partition the work of actually building the software. Yes, we figure out what data structures we need to use at a high level and then grow the code incrementally to tie it all together after that. But maybe that's fallout from working mostly in the system / embedded / network protocol space for most of my career where frameworks are not really a consideration.
I wonder where this notion comes from that you can not fail fast in engineering. Both science and engineering are all about fail fast and fail often or even all the time. This is the base tenet of both engineering and science. FAIL. A lot.
There was an essay that I saw on HN where the author talked to a bunch of "real" engineers -- i think the author had some 'real' engineering experience and software engineering experience -- to try and compare and contrast and see if the things people usually say about the differences are true.
I can't find it now! Which is frustrating me. Anyone else seeing htis have any ideas?
It was enlightening, and challenged some things we assume about differences. "real" engineering is also pretty diverse, of course, not just civil engineers making bridges and highways.
> Imagine if software was held to the same level of scrutiny..? Software is carpentered, some is designed, and little is actually engineered.
This is why I never approve criminalizing software bugs.
Imagine if you committed some codes, then turnout codes in that commit can be exploited by hacker 10 years later, are you willing to be imprisoned for creating codes that "fails to achieve a prescribed consensus of acceptability" ?
It's amazing how much this imagined interview matches up with my experiences working as a game designer. You could lift a lot of this, apart from the wild fame pieces straight from talking about game design where the designer is also heavily involved in implementation.
A world where design flair trumps actually, you know, working, meeting needs.
This is software turned into the movies, or architecture, where the valued work doesn't have to improve anybody's life, and accidents of birth determine rewards. Photographers flock to red carpets.
We already have too much of that here. Fashions dictate web page designs to the near exclusion of the actual purpose of any given site or page. Needed information is artfully concealed behind long downloads and meaningless animation. It is a plague.
I think people tend to forget about all the things in our world that are invisible, but keep our lives going. Like, does anyone actually care what the frame of their house looks like? Could they pick a fuel injector out of a bin of car parts?
Any attention to the visual appeal of these objects is wasted effort. Some things need to just do a job.
Often the particular choices made for "visual appeal" actively detract from correct function. Designers notoriously insist on sans-serif typefaces, despite their objectively lower legibility. Most phone browsers and messagers make it impossible to select a face where lower-case L and upper-case i can be distinguished, thus: "lIIlllIlII". (Maybe they even differ slightly in your view, but which is which?)
It amounts to contempt for users.
Stewart Brand reported that Architectural Digest readers almost unanimously rejected a suggestion that their annual awards incorporate a measure of users' satisfaction with the building. Contempt for victims of designer malpractice is not just common, it is built into institutions.
I'd like some more specificity on how exactly this is supposed to work. Ok, you are going to build a webserver, now what? I draw a bunch of flowcharts describing the code that doesn't exist yet?
If the situation is not novel, then this step is only useful for documentation. If this is uncharted territory, any spec with this level of specificity will be thrown out in seconds.
The conceit of this article reminds me of http://ngnghm.github.io/index.html which has the same conceit - examining programming practices in a different universe.
I've been both a professional software developer and a professional software designer, and I always thought the two jobs were more alike than they were different. What is design? Solving a problem the best way, given a desired outcome and a set of constraints. What is engineering? See above. The tool set is the main difference, but the tool set is the least important part of either job.
For a decade I've mostly worked on the kind of products that require operating in close proximity to mechanical, electrical and system engineers. It's thoroughly and consistently disabused me of the notion I see in software-centric areas that software is uniquely creative such that it's different than other "engineering" disciplines. They are just as - if not more - creative within their domains.
Software does have lower commercial expectations in the areas of reliability and quality, and that does seep into the practices you see in design and implementation. But I don't think it produces a difference in kind.
You have to question if you are dealing with an unbiased subset of mechanical and electrical engineers. That kind of interdisciplinary collaboration usually occurs in some sort of cutting edge field.
The typical mechanical engineer is calculating HVAC loads for commercial buildings, not working on robotics. In contrast Google is probably only hiring a mechanical engineer for work that’s the tip of the spear.
(In fairness this also goes the other way. The software engineers working on yet another internal corporate CRUD app are largely invisible to the typical physical engineer.)
I have a very diverse background in these regards:on the creative side I was freelance designer and have a MA of Arts, on the engineering side I design electronic circuits, program, work in VFX.
I guess many devs dream of expressing themselves with programming, just like many film makers, painters, artists dream of expressing themselves with their craft. But what many don't realise, is that after a certain degree of mastery, you know that certain topics demand a certain handling. It becomes less about you, it becomes more about that fascinating aspect of reality you are dealing with.
Whether that fascinating aspect is a certain mood you wanna express, a certain story you wanna tell or a certain problem you wanna solve and create a beautiful elegant solution for is not that different to me.
However: people who always make it about themselves instead of looking at the topics they deal with will rarely create things of lasting value, neither in art or design, nor in engineering. If you chose that framework because you thought it was cool, not because the problem you are solving demanded it, you are still learning and not creating the solution the problem deserves.
I have also worked close to electrical and mechanical engineering for a long time. Apparently you have not worked around rapid prototyping of robotics to think this [1].
The universe posed in this blog post is absolutely horrifying.
Starting off from the beginning: The last thing the world needs is more hero-worship. The idea of a "softwarenova" is somehow even more self-absorbed than "rockstar" or "ninja". It is completely antithetical to open source principles of collaboration, meritocracy, and community. We should strive to include as many people, and as many differing viewpoints, to the creation of software as possible. This is both in the name of diversity and inclusion as well as correctness and reliability. Creating people who are "above the common masses" can only hurt this goal. To the (hypothetical) interviewee's credit, she doesn't like the term.
Secondly, this idea that things in our universe are somehow prematurely over constrained, while in the design universe you work with design briefs and things are looser. The problem in the real world is that things are too often underspecified - Either in terms of latency, or availability, or security, or correctness, or even in what problem you're trying to solve. I find this extremely confusing to be coming from someone who has worked on F# and functional programming. Functional programming has such a strong emphasis on types, and eliminating classes of errors via types. I think if the author went to work in a software shop using C# or Java or any non-functional language he would instantly recognize how this loosey-goosey approach leads to software that is utterly unmaintainable.
Thirdly, while I do agree that a great number of problems are ill-defined, that is not because they are wicked. Wicked problems imply novelty and uniqueness, but the vast majority of software is simply executing some CRUD operation on a set of data. It isn't unique or novel, there are clearly correct and incorrect answers, there are alternative solutions available, etc.
My imagined universe is almost the complete opposite of this one, that is, instead of thinking of software engineers as designers, we should think of them as mathematicians. (Or at least, that is the mental model that would help the world the most). Software should be formally verified, and be proven to be correct rather than aesthetically divined to be correct. We should seek to maximally specify software, much further beyond the way the industry is currently working. We should overwhelmingly reject the notion that certain people have a gift to work on this, and others do not, and instead seek to bring everyone we can up to a level of technological literacy. We should standardize as much as possible to make software easier to use, rather than having this boutique approach. We should avoid thinking that age has anything to do with how well someone can program, young or old. In other words, the field of software development needs to grow up, badly. If the field could live up to the title of "Engineer" things would improve dramatically.
> When you're solving a problem, even as you get to a more technical level, you always keep in mind why you are solving it. I noticed that software engineers in your universe often ignore this. You fixate on some technical solution and then completely forget what is the context in which you're building it.
Excellent, you’ve just described a horrible engineer.
An article poor in both concept and execution. A painful read.
> You fixate on some technical solution and then completely forget what is the context in which you're building it. Good software designers basically treat all problems they face as ill-defined, even if they are fairly specific.
Oh my god, I nearly fell out of my chair. YES.
What we need are tools for software development that are highly flexible, easy to use, and safe. We need to put those tools in the hands of someone like a software designer from this blog post: A domain expert (and/or intuitive listener) who deeply understands the business, understands their customers' needs, and can hook up whatever logic is required themselves.
(And uhh, shameless plug, I'm actually working on a product right now that attempts to realize this dream. We're hiring platform engineers. See my profile for contact info/links.)