IMHO, writing software is actually an excellent example of an engineering task. The reason we see all these failures in software development is, in fact, that most software developers are NOT engineers. This is where all this pain comes from.
The role of engineering is to apply a theoretical model/representation of a physical fact, in order to achieve a desired result. The role of science is to provide these models with maximum accuracy.
The better a model is (equations, empirical laws...), and the better the engineer masters this model, the more accurate will be the expected results.
Programming languages, frameworks and patterns are scientific models, designed to represent the working of a computer. Bugs and unexpected behaviors in a program are due to a bad application of these models. If you know exactly what you are doing (i.e. you have a deep understanding/knowledge of these models), there is no reason for you to observe some unexpected failures. The more experience you have, the better you understand these models, and the more productive you are.
Of course this is just theory: in practice, things are getting quickly complicated and it is humanly impossible to predict everything that will happen - that's why testing exists. But my opinion is one would see much less failures in software development if developers were actually experienced engineers. Many developers today have no idea what they are doing, essentially because they are juniors and inexperienced. Unlike engineers, they are not trained to mentally play with the theoretical tools they use, so they cannot predict the results. And yes, at this point software development becomes more an art than an engineering task. It's like juggling without being able to predict the trajectory of the balls.
I think the difference is that software is engineering that isn't treated like engineering. For example, if a new bridge is going to be built, years and years are spent building models, testing winds, and understanding the geography of the location. Models are tested in wind tunnels. Dampers are added for earthquakes that may or may not occur. Every truckload of concrete is tested. Metals are designed for exact specifications.
When software is built, the first prototype becomes the production version. People don't think about the future. Algorithms aren't chosen they are hacked together by someone who may or may not understand the system from beginning to end.
The problem with software, most times, is precisely that it is not treated as engineering. We build software to be thrown away. We don't build it to last. This article from Dan Bricklin changed my perspective on software.
He argues that we should treat software more like engineering projects and design and build it to last.
Furthermore, we require civil engineers building bridges to have degrees in civil engineering. For software, anyone can create it. People who never went to college even. But you'd never allow someone with only a history degree to design a skyscraper. We've become accustomed to software written by people who are not engineers. We don't spend the same kind of money on it and the users of it don't really care, because unlike a bridge, their lives don't depend on it not crashing.
If we treated programming like engineering we would finish one project every twenty years or so, and proceed to work on the next for the next twenty years.
There's simply no such counterpart in physical engineering to the iterative programming process. Engineers know how to build a bridge, that's why a particular design succeeds and is then followed by construction properly carried out. Programmers face unknown and unsolved problems all the time; in fact, a solved problem is not worth the programmer's time. A solved problem has already been implemented several times and can be reused, either literally or conceptually.
Engineers have been trying to solve the problem of a practical electric car with acceleration, velocity, range, and reload time comparable to petrol engines for decades. New things take time.
The reason we can write complex software in a few years instead of decades is that in software world, luckily, the cost of building, compiling, and trashing a prototype is negligible. Conversely, the complexity of projects has been upped a corresponding notch but only to the extent that we can still actually finish some projects.
Actually as someone with degrees and experience in both fields, you face unknowns FAR more often in the physical world than the software world. Man if only I had a step debugger and some logs for that turbine that just exploded.
There are a lot of reasons for the difference in disciplines but saying that software is "unknown and unsolved" is pretty far off the mark.
If what you say is true, we'd have electronic voting. Why don't we have that?
Physical engineers face unknowns all the time. Building in unknown environments for example. New materials. Climate change. I can't even count the number of different designs for bridges, buildings, and cranes.
Yes, but at least you know you're building a bridge. Instead of, say, developing a game and discovering your users want a photo sharing site instead (flickr). That's an extreme example, but it's very rare to build software that doesn't deviate significantly from what is initially specified. You could never engineer in those conditions.
And once you build your bridge, you're finished. It just has to stand there and take the load. It doesn't have to be unpredictably changed over the course of it's lifetime to also function as an airstrip and shopping mall.
Even the simplest of real-world software products you typically need to grow and adapt - sometimes dramatically - over time.
"There's simply no such counterpart in physical engineering to the iterative programming process."
Not true! There are so-called handbook engineering problems -- change the ratios in this gear box -- that can be solved by a monkey with a textbook.
However design-from-scratch hardware engineering usually involves a great deal of iteration. The first phase of designing a new chemical or fluid system is a mini-research project to measure the performance curves of various possibilities under a wide range of conditions, and if none of them are good enough it turns into a blue-sky science project. Radio electronics projects commonly start out as a pile of tiny two-layer circuit boards, one for each subsystem, so they can be cheaply iterated in parallel, instead of having to repeatedly redo a single massive board. ("We have to spin (redo) the system board" not being a happy phrase to a project lead.)
In all the sensor and detector projects I worked on, the embedded real-time firmware was always the bright spot that was practically guaranteed to work.
Don't even get me started about designing research hardware for scientists ...
"We should treat software more like engineering projects and design and build it to last."
No, we shouldn't. If you build a bridge to last 200 years, it will remain unchanged for 200 years.
What piece of software can you possibly imagine still be useful if it remains unchanged for 200 years? 50? 10? Hell, can you name a piece of software that was last updated 5 years ago that you still find useful?
Most software becomes obsolete by the changing digital landscape within a year or two if it isn't updated to keep up with the times. That's a major difference compared to physical objects - the physical landscape doesn't change nearly as frequently. That said, we still don't build most physical objects to last 200 years except for the very largest of a projects - and we frequently demo small buidings to put up high rises as the city landscape changes.
Hell, can you name a piece of software that was last updated 5 years ago that you still find useful?
Yes, tons. For example, the firmware for my car's electronic fuel injection (EFI). A lot of software can't easily be updated, or lasts far longer than originally intended.
Treating software development as solely an engineering discipline is, in my opinion, a mistake. That being said, the vast majority of software projects today would benefit greatly from that approach.
Because of your obvious lack of knowledge about "real world" engineering. Think of the planning that goes into just simply replacing the guardrail on a bridge. How are you going to cut it off, replace it, and install a new one on a changing bridge while least affecting traffic? In the case of the bridge I drive over every day the answer was to remove each ~100'x100' section of road deck (there is one in each direction) and replace it in its entirety, one a night, for months. This was to widen the 5 lane bridge by about 3' and raise the guardrail by about a foot. Sure a "minor" revision but I guarantee you could have paid for Flickr and a pile of VC backed startups making a hundred different projects for the cost of having a crews of guys working for months, a barge crane, and thousands of tons of steel and concrete. It just isn't even comparable. Specs changing are a constant fact of life in real world engineering too, they just aren't the #1 issue that has to be dealt with.
I think you're illustrating my point perfectly. An absolutely minor change to replace a guardrail is more expensive "than Flickr and a pile of VC backed startups making a hundred different projects".
Engineering projects do not change frequently and radically. They can't afford to - but more importantly - they don't need to, certainly not at anywhere the pace of software.
A bridge may undergo some changes and facelifts over a 200-year period, but it's still basically the same bridge. But there isn't much software that could last even a fraction that long and not be completely obsolete (without most of it's code getting replaced). In the case of a bridge, the underlying geography isn't going anywhere. That's just not the case with software - even if the platform still exists, the rapid pace of technology means that the software will stop being of any practical use.
A piece of software needs to keep evolving, or it's dead. The same just isn't true of most 200-year engineering projects.
I've waxed eloquent about this on HN before, but some subfields of software development is rather treated like engineering (and, I would dare say, qualifies as engineering).
I work on avionics software. There is a lot of planning. There are formal requirements based on industry standards and the needs of the users. There are oodles of verification procedures to ensure that the software written does what the requirements demand. There are peer reviews. There is certification paperwork. There are quality audits on the requirements, the code, the verification, the reviews, and the certification paperwork. There are on-ground tests. There are flight tests.
We do require the "engineers" to have relevant degrees, and if our software crashes... well, hopefully nobody dies, but, it's a real and sobering possibility.
On the other hand, all of the engineering-esque work that goes into avionics would likely be utterly wasted on most web applications, iPhone programs, and video games.
> If you know exactly what you are doing (i.e. you have a
> deep understanding/knowledge of these models), there is
> no reason for you to observe some unexpected failures.
It's not bugs that doom projects, it's misunderstandings.
Most software fails because it doesn't meet it's users' requirements. Because there's so many possible ways to write a program, and it's impossible to get an upfront specification that captures all of the needed detail, we end up building things that aren't fit for their intended use.
This is a big difference between us and other engineers. In most cases their requirements are well-understood by both the builders and the users. Build a bridge from X to Y that can carry 20,000 cars an hour, that will withstand a 9.0 earthquake, etc. That's why the models are so key, their job is figuring out how to make objects to meet those requirements.
Software's advances come through either better methods of capturing requirements, eg agile programming, together with better tools that make it easier to prototype and iterate on a product to figure out what users really want.
> In most cases their requirements are well-understood by both the builders and the users.
Software has the luxury of the ability to have rapid change in the hands of users. Web startups are a good example. Even desktop software is a good example. Generally you don't want to apply a strong engineering model to them because the engineering process is costly in time and people. Software on a submarine or for the disk brakes on a car is not a good example because the rapid change can't be cheaply and transparently pushed to the users. Those systems require engineering.
For material products, show me examples where one does not HAVE to get upfront specifications. How would you release early and iterate on an MP3 player or a stove or shoes? The exceptions I can think of are food and landscaping.
Systems (material and software) are engineered because they HAVE to be engineered, not because engineering is inherent in material products and absent in software products. If the system does not require engineering (for safety, cost, risk) then it doesn't look like a good candidate for engineering. Do you engineer your dinner? I don't but NASA does for their astronauts.
"they are not trained to mentally play with the theoretical tools they use"
That's because the actual theoretical tools that are in place from computer science are generally not practically applicable to 99.9% of the work that most developers do.
A knowledge of lambda calculus, CSP, Turing Machines etc. etc. is of no relevance whatsoever to someone building a CRUD business application or (for that matter) another client for Twitter. Formal abstractions from other fields aren't of much use because the mathematical techniques used for modeling the "real world" don't apply too well to programs.
"Programming languages, frameworks and patterns are scientific models, designed to represent the working of a computer"
That an interesting idea - but as far as I can see, completely wrong. Scientific models are created (usually using maths, but not always) to provide logical structures that have predictive and explanatory powers - and have to be consistent with observed behavior and have be falsifiable.
Programs aren't constructed to explain observed behavior of computers or to predict what computers will do - they are created to control (through layers of abstraction and translation) what the devices will actually do. So I don't think "scientific model" is the right phrase to use.
"A knowledge of lambda calculus, CSP, Turing Machines etc. etc. is of no relevance whatsoever to someone building a CRUD business application or (for that matter) another client for Twitter."
Fair enough. These are used as scientific materials for scientific purposes, and are indeed not suitable for an engineer's work. However languages and frameworks are abstractions (= models) of how a computer works and engineers do use them to control it and get the ordered results.
Think of a computer as a black box. You want to get some output from it (calculation results, commands, whatever). You need to know what input you must give them in order to produce the expected output. This is were programming languages and frameworks come in. They offer an abstraction of the working of the computer that you can use. Programs are indeed not constructed to explain how a computer works because they are the input, not the system itself. They are the input a developer needs to provide to the computer in order to have the requested result.
How does a developer know what input to give (i.e. what code to write)? He has to understand how the system works, or rather, how the abstraction layer/framework he's using works. If his knowledge is not perfect, he might observe some unexpected results. He then needs to investigate what part of the framework he did not understand, and adapt his program consequently.
This is what all engineers do. They manipulate a system using an abstract representation of it, they gain experience by figuring out how or why the output they receive is not what they expected and by iteratively adjusting their understanding of the system.
Since all engineers do this, why would developers do things differently? Engineers learn to manipulate abstract models (in a lot of different domains), and IMHO many developers fail because they lack such training.
> Since all engineers do this, why would developers do things differently?
Because, as arethuza said, programmers don't have the tools that other engineering disciplines have. Most notably, the equations that allow for predictive analysis and proofs of correctness.
A civil engineer that builds a bridge can be fairly certain it won't fall down even when it's still on paper, because he can reason about the forces and stresses and even environmental degradation via well known equations and models. This collection of predictive and proof techniques is what makes it an engineering discipline.
Programming doesn't have that, at least not yet. As you point out, the program is the input to the machine, not the predictive model. But at least if the machine acts deterministically, we can focus on proving things about its input, the program. There has been quite a bit of effort in the research community to come up with ways to prove correctness, but the state of the art is still a long ways from being something actually useful. Functional languages are (partially) a reaction to the prediction requirement. By not having side effects or depending on state, it's easier to make predictions about behavior. (Of course, lazy evaluation buys you a lot in expressive power and data structure efficiency, but offers a new hurdle in predicting performance).
And if you really look at it, Software Engineering is, well, not. Everything I have seen in the emerging Software Engineering field is really about managing people. It's about how to get 1000 developers on the same page (which is like herding cats) so they can crank out an operating system. Sure, there is a lot about testing and validation and change control, but it's still mostly from the angle of controlling the risk of a developer doing something stupid. The tools that would allow a software architect to write an exact specification of a project (bridge blueprints), prove it'll meet all requirements (not fall down), then turn it over to junior programmers (construction crew), just don't exist yet. Check out Rosetta for an ongoing attempt.
So maybe someday programming truly will be an engineering discipline, but right now it's more like having a million Wright brothers all building their own airplane, using whatever materials they are used to working with, rather than an established discipline like Aeronautical Engineering.
Your understanding of software development is limited.
Most real-world software projects aren't the kind of straightforward batch processing jobs you might have done in the mandatory entry-level college programming class that they make engineers take.
They aren't about trying to figure out the input (code) to get the 'expected output' you want.
They are about dealing with user interaction, and users - messy and unpredictable. It's not merely a matter of looking at the possible 'inputs' a user can take and trying to explore all the possible stateful interactions; if the software doesn't do what the user actually needs, or if he can't figure out HOW to make it do what he wants, it's a failure. Figuring that out is called design, and there's no way to 'engineer' that.
And physical engineers don't have users with even more complicated interface problems? Most computer systems are not real time. Airplanes, cars, factory processes, cranes, power plants, even chairs have an interface with users that is more complicated to design.
The people who make chairs aren't engineers. They're designers. Industrial Designers.
The people who 'design' the look and user interface of the car? Automotive Designers. Not automotive engineers.
The people who 'design' buildings are architects. They work with engineers to make sure the buildings 'work'.
Software is most akin to architecture, except the role of 'designer' and 'engineer' is rolled into one. There's only a portion of the the development process that you can actually apply engineering principles to.
I disagree with what you are saying. For each item you mentioned that is irrelevant, the same is true in electrical engineering or mechanical engineering. It should be obvious why a Turing machine isn't relevant to modern programming. It should also be obvious why Einstein's models of the universe aren't relevant to automotive engineering.
There are many computer science principles that are relevant to even a CRUD business application, encryption, hashing, caching, parameterization, abstraction, design patterns, and more are all applicable to problems for those kinds of applications -- they're just rarely used.
Sorry, I wasn't meaning that there aren't useful things in CS (I'm a CS graduate myself). What I meant was that the techniques for formally modeling software so that you can reason about it aren't as well developed as they are for normal engineering subjects.
Which normal engineering concepts are you saying are more formal than in CS? I had to take a "Formal Methods" course in college. Gears are loops. Switches are switches. If statements are transistors.
Also recognize that CS is a much newer science than physics. Some of what you are saying may be due to the immaturity of the science, but it doesn't mean that it isn't engineering.
I took a lot of other engineering courses including statics and dynamics, thermodynamics, electronics and courses for premed requirements and I don't see a lot of difference between computer science/engineering and biomedical engineering or civil engineering except our perception of of the physical engineering as more important -- due only to their visibility.
CS/CSE is hidden from view. People don't understand where while loops are operating, but they can see gears spinning and cables holding up bridges.
Compare typical software projects to medical software products as a niche. Much more design, time, and money is spent there than on Web 2.0 sites because lives depend on it.
I phrased that incorrectly - the techniques are all there in CS, but if anything they are too formal. It's the practical benefits of applying formal methods that are missing from general development (usually for good reasons).
A while ago I had an argument with a chemical engineer about exactly this. He claimed that software engineering is an immature field and that we will eventually learn how to apply engineering principles to get more reliable programs.
When I countered that you mostly don't even know what you're modelling and that the design state space is much larger in software, he retorted that all sorts of things can go wrong with just a single pipe in a chemical plant but that it doesn't stop reliable plants from being built.
But he didn't get it: that pipe is a component in a well understood framework. It's nothing like the unexpected things you get in software. The software equivalent is that of a buggy component whose specification is well understood - if you rewrite the component, you'll get what you expect. In software, proper specification is the rub.
I have worked with many electrical and electronic engineers (I studied CS) and not one of those engineers was on average better at writing reliable software. They did try hard to specify their systems in advance, but the actual systems soon started diverging from their specifications.
I grant that the absence of evidence is not evidence of absence, but it is equally wrong to keep maintaining in full faith (and without convincing evidence) that software development is an engineering discipline.
I have never seen anyone argue convincingly for nor against the engineering approach in software. If anyone has done a study on this, I would love to read it.
By your own definition, software clearly isn't engineering. Most programming languages aren't representations of a computer at all - in fact, one can argue that programming languages have become more powerful the more they've distanced themselves from the underlying working of the computer.
But it's the "in order to achieve a desired result" that your argument really falls down. Engineering works because the end result can be clearly specified - that's almost never the case for software. The goal of most software is to make the users happy. And there's no way you can engineer happiness. Instead, you're faced with a design problem - and design is by it's very nature is iterative.
As software development has gotten more iterative, it's also actually gotten more effective. In fact, it seems like the more we embrace that software ISN'T engineering, the better off we are.
The role of engineering is to apply a theoretical model/representation of a physical fact, in order to achieve a desired result. The role of science is to provide these models with maximum accuracy.
The better a model is (equations, empirical laws...), and the better the engineer masters this model, the more accurate will be the expected results.
Programming languages, frameworks and patterns are scientific models, designed to represent the working of a computer. Bugs and unexpected behaviors in a program are due to a bad application of these models. If you know exactly what you are doing (i.e. you have a deep understanding/knowledge of these models), there is no reason for you to observe some unexpected failures. The more experience you have, the better you understand these models, and the more productive you are.
Of course this is just theory: in practice, things are getting quickly complicated and it is humanly impossible to predict everything that will happen - that's why testing exists. But my opinion is one would see much less failures in software development if developers were actually experienced engineers. Many developers today have no idea what they are doing, essentially because they are juniors and inexperienced. Unlike engineers, they are not trained to mentally play with the theoretical tools they use, so they cannot predict the results. And yes, at this point software development becomes more an art than an engineering task. It's like juggling without being able to predict the trajectory of the balls.