data.

http://embeddedgurus.com/barr-code/2010/09/the-sad-state-of-...

I was in a class taught by this guy and I seem to recall him saying that a large (20-40%) of embedded developers he surveyed reported using no version control software.

and look at some of the responses in that thread!

"I think that gcc -Wall -Wextra can be subtituted by lint tool (if you use gcc)."

Yeah, but that's a vendor issue; it's not that the Engineers don't want to use version control.

We'd like to, for example, but we usually can't. All of the file formats for these PLC tools are binary blobs, and since the language is usually a graphical language (ladder logic) the conventional idea of diff is pretty meaningless unless someone writes a proprietary version control to handle the binary blobs + representing ladder logic through diffs.

We still try to use it, but we lose most of the benefits. I suppose you still get a changelog, but you can't ever merge branches since merging binary blobs with binary blobs will not go well for you.

That's pretty valuable, even without the ability to merge.

The exception of course is some very large companies that do everything in house.

You never change the SW without changing the hardware?

Wowsers. No new features and no bugs.

that's obviously bad, but is it higher than developers across disciplines? i'm sure you guys all use version control, but if you surveyed all web devs, how many of them use version control? i'd bet its a similar number

Our customers do not, in general, have any of those things. Many of them have no central version control (independent teams sometimes have subversion setup for themselves). Those who have it are usually using CVS, and usually don't use it very well.

Embedded is a field of "old fogeys" still. It isn't taught in school, it isn't advertised very well, and the companies who do it don't offer the cool perks that web companies do. A lot of these companies are running into problems where a large portion of their engineering force is retiring, and nobody is available to replace them.

This is more likely to be the case with our big customers than the smaller ones. The small ones usually have at least have some people pushing for modern project management, though it's often not implemented across the whole company.

Many of them got into the field when microcontrollers were extremely limited and C compilers were unreliable -- most of them learned assembly and used it for most of their career. There is still a tendency for the older ones to not believe us when we say their project can be written in C. Ten years ago, they would have been right.

Obviously, you can't generalize an entire industry as being "way behind", but many embedded firms just haven't progressed the same way those in other industries have.

Also, it's critical to realize that it has nothing to do with the engineers not being smart. They tend to be quite brilliant, in fact. But many of them are electrical engineers with a background in analog electronics, physics, and math, instead of modern software development.

A bit of personal experience, stuff like Stuxnet as well as seeing bits of security research that confirm that notion.

Besides, not all control engineering is done by large corporations. There are tons of customizations done by smaller companies, whose competence varies.

Mind you, I do live in the Netherlands, and it seems lots of PLC systems are designed by electrical engineers who started out in that or something else and don't know about much on the Joel Test. But I can be proven wrong.

A contributing factor is probably that each vendor has his/her toolkit which in some sense is proprietary, and so the market for general tools is smaller, and it's an enterprise market making it harder to sell to.

That's probably true, given that PLCs were originally for electrical engineers as means of having "software relays". The concept of computer engineers came much later, and for computer science, they typically have zero electrical background which means that even if they program really well, it can be the case that they don't understand the physical implications of the code that they've just written.

In computer science, writing True might mean it's magically on, but in real life, a valve actually has a small delay before it physically is open. So a time delay has to be placed on the output. That's a simple example, but there are other such nuances of electrical/physical devices. Also, much of PLC programming (especially in process control) is control theory (PID controllers, essentially differential equations so that devices actuate "smoothly" instead of in a "jerky" manner). This also seems to not be covered in a CS curriculum.

The industrial controls industry has a very 1980s/1990s attitude for cybersecurity. I don't envy the public-facing engineers working for the vendors.

I'm always reminded of a conversation with a fresh Google hire wherein she said "How can you stand working at a utility, don't you want your work to affect PEOPLE?", as if utilities don't have hundreds of thousands or millions of customers that we provide an essential service to.

I think installing all of the Siemens software requires something like 15 reboots, at last count. And I have no idea why.