I see a couple of other issues, which share a root cause.
1. A lot of research is based on mining open source projects which bear little resemblance to how software engineering is done behind closed doors.
2. Very few prospective trials (much less randomized controlled trials), which are considered the gold standard of evidence for a good reason.
These retrospective studies are very useful, but to really have faith in a conclusion that X is better than Y you need to assign several engineering teams to condition X and several more to condition Y and then compare.
Nobody has money for that and it would be hard to find even one team (let alone several) that is going to let a local professor flip a coin and decide whether they are going to do Scrum or Kanban. You'd also need quite a few teams because there would be so many uncontrolled aspects.
Researchers in this area are in the very difficult situation where the interesting things are difficult to measure and the things that are easy to measure aren't that interesting. Somehow they still do good work and that's very impressive but I don't envy their task.
I think the whole "you need too many subjects and too much money" objection to doing science is very weak. There are other fields of research where they find inventive ways of experimenting on a shoestring budget (I think some social sciences belong in this category.)
You can even do proper causal analysis when all you have is an observational study, given that you can get an expert to produce a decent guess at the causal network. (And this guess can to some extent be verified by plain correlations.)[1]
Sure, you might not get the extremely high external validity of e.g. particle physics, but in many cases something is better than nothing.[2]
Heck, screw external validity. I'd be ecstatic to see more experiments just within an organisation, focusing on internal validity. Every engineering organisation can afford to experiment with different tools and practises. There are huge productivity gains to be unlocked (is my hypothesis, anyway), and the cost is small.
If we get together enough small studies lacking external validity, but which point in the same direction, we might even be able to infer some external validity between them.
[1]: For more on this approach, see Judea Pearl.
[2]: To use an example close to my heart: pregnancy is a condition where you rarely see randomised trials, and any research is sorely underfunded. Yet when people actually take the time to sit down and answer questions with data, we get something meaningful out of it.
I wouldn't be surprised if a few organizations had run major experiments over the years, with results never seeing the light of day.
It would also be great if there was more awareness of the kind of data the research community needs so that if someone saw a natural experiment shaping up in their company they knew who to call.
Regarding budgets, these researchers are a lot smarter than me and I bet they are doing their best with what they have. What might be good though is having several labs join forces and collaborate on one very good trial of a core issue.
Then you have a hope of settling a question and moving the field forward while creating some infrastructure that will help answer other big questions.
Maybe that's happening, I haven't seen a lot of it though.
It's worth pointing out that this blog post was published three years before Accelerate, by Forsgren, Humble, and Kim. That book demonstrates what you can do just with surveys. It's not about software engineering per se - it's got a devops focus - but the overlap is significant.
As far as I'm concerned that book significantly changed the game, not only because of the findings, but because it showed that hard results are possible in this arena. The downside is that you need enough data to make the maths work, and that precludes applying the techniques to internal studies in most organisations. There's just too much noise.
Behind closed door software engineering is actually done only when the bad first implementation runs into limits. Software Engineering is similar to JIT-Compiling, it only applies to the hot-loop in a company.
Things that must work, and were every transactional loss has a cost to it. The first fb-implementation could be lousy technological wise, but the moment it generated add revenue and the site was "slowed down" to the point it lost people, cooperated software-engineering starts to exist.
Any architecture created before these constraints applied, is just legacy and if burdened with useless abstractions, often even a total loss. So, the war stories, the strange Optimizations needed to get something valuable enough for somebody to be "on call 247" thats the actual software engineering research.
Academia does not really work under these constraints, thus its solutions are "beautiful" but only relevant, when "pre-incident" architecture proofs it can survive even to post-incident.
You can do trials, fairly cheaply, if you do them on CS students. But then you're (almost always) working on smaller code bases, and always working with very inexperienced devs (by professional standards). So the applicability of the studies to the real world of working software engineers is... questionable.
In particular, if you have a study showing that X tool helps people in Y situation, and your subjects were CS students, then you have a study showing that X tool helps CS students in Y situation. Does it help people with 5 years professional experience? 20 years? You can't tell. It might, but you don't know.
The use of students in place of professionals would have gone as #3 in my list if I had thought of it, thank you for adding it.
Generalizing from studies where all participants were students is one of the things that psychology and the social sciences have had to stop doing (except when they are studying students specifically, of course).
We are currently growing into it and may find we get the same mixed results.
I work with a lot of undergrad CS students and the difference between an undergrad senior and a second year junior developer is enormous.
That's not a problem in general, but it is if you want to test software engineering practices with students beyond just working the kinks out of your protocol.
It occurs to me that you might be able to actually run real tests, with real engineers.
Say you're in Iowa or Kansas or somewhere. You can probably hire software engineers, with 5 to 10 years experience, for $100K/year, maybe even less. Hire 10 of them, for two years. That's $2 million.
Assign them to random teams. Give them non-trivial projects that run, say, three months each. That gives you eight experiments you can run. Change the methodology, or the language, or whatever you're trying to study.
You say that you can't get funding for a $2 million experiment? Tell Microsoft, and Oracle, and IBM, and the federal government, that the things you learn will enable them to more efficiently create software. See if they'll fund at least part of it.
Many "engineers" don't know what a hash table is under the hood, what binary search is or what pointers are, but they know Framework X, or Tool Y. They lack curiosity, desire to learn how things work, desire to discover. They don't trust concepts unless those concepts have a relatively large following and are deemed as "fresh", "cool", "new" and "inspiring" by trend setters. Also having some popular articles on Medium helps with trust.
Presumably they are never going to have to implement or modify a hash table (which kind? Cuckoo, Hopscotch, Robin Hood, ...?), but always use one provided by a library (very likely the standard library of their programming language). Of course knowledge about the performance characteristics of a hash table (especially in comparison to lists) is useful.
Binary search isn't all that great due to cache misses. It's likely you won't regret using a hash table instead.
As for pointers, depending on the programming language you use they may not be exposed in a way that the knowledge is all that relevant. In a language such as C you can't reasonably avoid them, but for Java, or Python, SQL, or JavaScript you just have references, which tend to be much simpler.
Knowledge about how to use Framework X will probably turn out to be a lot more relevant and helpful for the day-to-day work of most developers.
It's funny, when it s not the devs, what you describe sounds strangely exactly like what many ivory tower CTO are like, "researching" useless shiny craps by reading some headline like "google does monorepos" and coming back saying "but you devs are just mindless morons, look at google they do monorepo, you should have the curiosity to merge all our github stuff into one giant repo".
It's more interesting and useful for me, and I assume other people, to learn about new ML architectures or frameworks that will facilitate my development than the underworkings of a hash table. This will probably bite me in the back eventually, but I see a lot of opportunity cost when choosing what to learn.
> I’ve referred to empirical studies that strongly suggest the adoption of particular practices, but experience, anecdote, and context have usually won out over evidence
like...what? which practices?
> I’ve demoed exciting tools from research, but my team has found them mostly useless, since they aren’t production ready.
just getting teams on board with "classic" tools like static analysis and sanitizers is major. And those can usually be described as 'production ready'
The guy clearly does it the wrong way, he has a solution and everyone is looking at him "but what problem do we have this solves".
So yeah, I mean even if these tools were production ready, they wouldnt be used. If they solved a problem people had, you d bet we d stay at night to make them production ready.
He seems to think his devs are like mindless zombies needing constant intellectual feeding from him, the researching CTO, but it s probably more like they raise their eyes "oh god what is the guy producing no value day to day going to show us again while we stress on the deadline pretending to listen to him"
If you compare say diabetes abd perception of time (another post), you can see the issue of research in many topics. Study need to be relevant and on the ground. But if someone study emacs, vim vs vscode or ispf can this be even master thesis material.
I think part of the issue here had to be that the it’s very difficult to produce robust, actionable results about complex systems of humans. Research (by small teams at universities with limited resources, looking for minimal publishable units, typically) isn’t as effective as it might be for topics where isolated experiments are more possible.
I was a software engineering researcher (Ph. D. and everything) before I actually became a software engineer about 20 years ago. I became one because I realized it was becoming quite awkward that I wasn't one. There was more than a bit of impostor syndrome.
Software Engineering as a science has always been a bit controversial; mainly because a lot of computer scientists are involved with it and it is a bit of a soft science that most hard beta types would be uncomfortable with. Additionally, the whole notion of software engineering actually being an engineering discipline is not something everyone agrees on either.
Software Engineering is fundamentally a process that involves people interacting. That means a lot of the challenges and problems are related to that and a lot of the solutions are related to that as well. Simply put, a tool won't fix your process and people issues, generally.
The software engineering challenges in the industry are numerous. But I'd say the number one problem is that the demographics are completely skewed to propelling very inexperienced individuals into leadership roles because there simply aren't enough more experienced people around. Basically, the number of programmers doubles every five years or so. I'm 47; I routinely work with engineers half my age. Having anyone else in a team even close to 35-40 is very rare for me. The reason is not that my generation chickened out and went off and did something else; it's because we are outnumbered by about 2^5= 32x by people aged 22 or so. And about 16x by people aged 27. And about 8x by people aged 32. The median age in this industry is is 25-28. If you are lucky those people actually finished a degree and had one or two meaningful jobs before they joined you on your current project. Looking back at myself at that age, I definitely wasn't very senior.
But it's great fun to work with young people and it keeps me sharp. The reality is that young people don't have a lot of older peers to learn from. So, a lot of issues in this industry are repetive. Young person gets propelled into a senior role and gets to rediscover all the same issues that people were dealing with 30 years ago as well. And they get to make all the same mistakes before they figure out some solution. The smart ones, pick up a few software engineering books and find out that people were dealing with the same shit 40 years ago and found some pragmatic ways of dealing with stuff.
However, I'd caution anyone to apply anything they read blindly and dogmatically. Very little of it has been empirically validated in a proper software engineering research study. Doing so is hard and requires a lot of work. Most of the software engineering books out there don't bother with doing that work. That doesn't mean it's wrong but just that you need to separate facts from opinions. If it has the word "Agile" on the cover, you might want to keep that in mind. Most of that stuff is written with the same level of enthusiasm and assertiveness as the stuff that came before that.
The fundamental problem is the same as with business and management research - any useful signal is completely drowned in an ocean of noise. Success of a project depends on the characters of the people involved, connections of the project lead, market timing, sources of funding, luck, current phase of the business cycle, past project experience, marketing and hype, sales teams, relatedness of icons and animations, etc. - more so than on any technical or organizational factor. I've yet to work on a project where technical reasons were at the root cause of the lack of success
You're looking in the wrong place. It's a sort of survivorship bias.
You might not have worked on an individual project where technical reasons were the root cause of failure, but I've seen many projects not happen at all because changing a badly engineered platform was deemed too risky or too expensive, or just couldn't get scheduled because another project - made slower by poor past engineering decisions, but important enough to go ahead anyway - was occupying the team's time.
Focusing on projects and not on platforms and products creates these blinkers.
Couldn't it be that it is much more signal than noise? It might be too many and too densely interconnected to disentangle (yet) though. Being technical people and used to clear cause-relation analyses we might just give up and call it noise?
How does one follows research on some specific topic? Say I am interested in verification. I am constantly hitting paywalls for research papers that I have absolutely no idea what the value is. Let's be honest, even those papers are amazing, as a self taught developer, no way i could extract every possible valuable information from it. This also means that I have no access to some university account where I could download papers "just to see what's going on"
Research looks like a hermetic world and little osmosis happens with surroundings
The reason there's little research is because there is little money and structure in STEM education in general. I have been to multiple name brand uni's that simply do not have basic lab equipment. Some high schools had more equipment and expertise in software than uni's did.
Why? Because education as a business has fundamentally broken incentives. Most educators have never worked in the private industry, or did so for very little time. This means they don't actually know what they're talking about, they just follow courses accreditation allows them to. You will never make as much as a professor in software than you will privately, unless you are in one of the top schools that are inaccessible to the overwhelming majority of the population.
The workplace politics are egregious, because universities are not a meritocracy, universities have created a system where your skin color matters a lot more than your qualifications to conduct your job. For example, someone becoming head of department because they're from India, not because the product of their research or overall conduct in their job was the superior choice. This trickles down, students become the professor, and over time we are now here.
Tech companies are the wealthiest in the world out of any major, yet contribute very little funding. If anything they make a killing off of schools and students buying hardware and software. Look at a company like Apple, that goes out of its way to make exclusivity deals with university chains, yet still charge their artificial $1200 price tag for the mandatory macbook. These students are having the life chocked out of them, why would we expect them to want to keep doing that and do "research"?
1. A lot of research is based on mining open source projects which bear little resemblance to how software engineering is done behind closed doors.
2. Very few prospective trials (much less randomized controlled trials), which are considered the gold standard of evidence for a good reason.
These retrospective studies are very useful, but to really have faith in a conclusion that X is better than Y you need to assign several engineering teams to condition X and several more to condition Y and then compare.
Nobody has money for that and it would be hard to find even one team (let alone several) that is going to let a local professor flip a coin and decide whether they are going to do Scrum or Kanban. You'd also need quite a few teams because there would be so many uncontrolled aspects.
Researchers in this area are in the very difficult situation where the interesting things are difficult to measure and the things that are easy to measure aren't that interesting. Somehow they still do good work and that's very impressive but I don't envy their task.