You're right, and more generally all of the issues are related to the fact that scientific incentives don't typically align with good development. At the end of the day, over a period of 3+ years, I'd rather have the results of 3-4 software engineers compared to 10-12 grad students. However, for <3 years, I'd choose the grad students. Pretty much every incentive in science (e.g. grants, awards) prioritizes being prolific over a short period of time.
Here in Oxbridge there's pretty generous funding for hiring software engineers to maintain some biotech software, and much lower pressure to publish. I know many open positions. And they tend to fund you for very long time. I know people working on said positions for more than a decade. Sometimes 20 years.
The problem is more pay gap with industry. Even though they tend to pay well by academic standards (e.g. no PhD required, yet pay is much higher than a senior postdoc), the salary is still way below what industry in London or Oxbridge offers.
Furthermore, you tend to be surrounded by non-technical people which may be tough in the long term. Nobody appreciates what you do. Not even your boss, who may know zero about computers.
The bottom line is that positions end up being vacant for long time and tend to be filled by biologists with a bit of coding experience, underqualified IT people or, rarely, really competent individuals that want a break from industry.
> The problem is more pay gap with industry. Even though they tend to pay well by academic standards (e.g. no PhD required, yet pay is much higher than a senior postdoc), the salary is still way below what industry in London or Oxbridge offers.
I know sufficiently many good programmers who would love to do scientific programming (because they love science) and would immediately accept less pay.
The problem, in my opinion, rather lies in the non-monetary work conditions. For example, in Germany it is nearly impossible to get permanent employment contract when working at a scientific institute and doing something remotely related to scientific work. Even worse: you are not even allowed to work more than 6+6 years (before and after doctorate) in a fixed-term employment position at a scientific institute If this time is over, you are not even allowed to take any non-permanent contract at a scientific institute (the infamous/insane Wissenschaftszeitvertragsgesetz (WissZeitVG)).
No programmer (even if (s)he has a great passion for science) will be willing to work under such extraordinarily bad conditions.
Here it is a bit better. Most positions I know of are de facto permanent. On paper they are not, as most labs go through 5-year funding cycles. So there's a tiny chance of loosing funding. It's quite rare for big labs.
Besides, places like MRC have created permanent research assistant positions. Which are actually permanent and put zero pressure on publications.
As you say, connecting with interested and talented programmers is another problem. I feel that Nature Jobs postings, which were already a big leap forward for rusty uni administrators, are not good enough.