Well, then science needs to do more to create an environment where software engineers would want to be an RSE.
From my experience "science" does not appreciate RSEs, the compensation is bad, the freedom is bad, and you get all of the big company bureaucracy from the University.
It's just a bad environment and to top it off, even if you have a PhD many will see you as "less than."
There is simply no incentive for science to do proper software engineering. It does not directly produce papers or money, so nobody does it. I've seen it several times in my own career in several countries.
PhDs having their own version of the code, incompatible with the one of the Postdoc sitting opposite of them. Each slaving away at their niche project but nobody there to bring it all together. This particular university had really great code and they thought they could sell it. But they did not even use git and when I gave a presentation there about it, I was met with absolute rejection. They don't have an idea on how much effort it takes to maintain a commercial code, unit tests, customer support etc.
I'm now working at a company that produces simulation software spun out from university. Fantastic job and we do all of those things mentioned above. But obviously my paper output has been near zero even though we sometimes do cutting edge research.
I wish I had a solution to the problem as so much grant money is wasted that produces a paper or two and the corresponding software for that just goes to digital nirvana, it's a real shame.
This isn't completely true. My wife is a postdoc at a medical lab at Stanford and there's a big drive to push good software practices - containerisation, tests, documentation, etc... and a good friend who's a postdoc in medical imaging at Oxford runs training courses for their labs software (along with following all standard engineering practices you'd see at a company). This will be subjective of where you're working and if you're driven to write good software. With the reproducibility issues in science, writing good clear software is becoming more important.
Of course my statement wasn't meant to be absolute. I come from an engineering background. I am not surprised that in a medical setting the situation is somewhat different, after all I guess the stakes there are higher. If your CFD code does not perform well, who cares. If your analysis of some medication is crap, chances somebody cares is going to be much higher.
Also from experience, it helps if universities have closer ties to industry. As an industrial player you can't work with software that does not deliver reproducible results.
I'll agree that there is increasing emphasis on reproducibility and _useable_ software in academia. Writing documentation, unit tests etc. is still not really rewarded properly, but at least within the current paradigm such efforts are often rewarded with more users (and therefore citations) which is rewarded. Soon, hopefully, it'll be recognised more directly.
Also, I'm currently a postdoc in medical imaging at UCL, super interested in learning a little more about the group in Oxford you mentioned if you're OK with sharing a link/group name? I may be able to guess but just want to check!
RSEs should not be working on, or assisting, research at the PhD level. By that I mean news/skunkworks stuff. They should be taking the output from a PhD and turning it into core software for the group. That doesn't solve the collaboration issue between PhDs/Postdocs, but there is a particular point in a research project lifecycle where it makes sense to hire an RSE.
A bigger challenge is that most PIs are not project managers and have no experience as such, so they don't know how to express what they need in a structured way, or how to steer their group to collaborate properly. Outside computer science, many would struggle to budget software development or compute properly on a grant application (and the assessors have zero idea either).
RSE's can and often absolutely should be involved at the PhD level. In my experience, collaboration between the scientist and engineer in the process of research iterations almost always produce better results. Each has insights the other may not, likely leading to better outcomes for the research, final product/tool and time taken.
The scientist just wants to focus on their research and once they have a barely working proof of concept, hand it over to the engineer to figure the rest out. The engineer wants a well specified design and prototype that they can lightly refactor to clean up, scale up and turn into a product/tool.
The reality is that approach makes it way harder for both, though most often harder for the engineer as they are generally at the end of the chain in Academia and have little power. For example, the code or spec from the scientist is often terrible, so the engineer needs to start from scratch and keep going back to the scientist to spec out the design as they were not involved at any stage prior. They may even find edge cases or flaws the scientist had not considered that are fundamentally problematic to turning it into a viable product/tool.
This is why the big corporate/industry research labs often have high level RSE that are involved in the research process and get their names in papers (they sometimes have PhD's themselves). They are not optimising for the scientists time, but for the companies resources
Yeah let me be clear. PhD students absolutely should get guidance from experienced engineers (so I was a bit over-zealous with "assist" in my parent post). But this should be more like understanding best practices, and they should feel free to ask questions and figure out how to write better code. There are initiatives to do this called Software Carpentry.[0] However, RSEs should not be writing code for students doing PhD level projects in my opinion, for exactly the reasons you mention.
I know some of the big research councils do this in the UK. For example STFC has a program where they'll work with universities and companies to production-ise research code.
> The scientist just wants to focus on their research and once they have a barely working proof of concept, hand it over to the engineer to figure the rest out. The engineer wants a well specified design and prototype that they can lightly refactor to clean up, scale up and turn into a product/tool.
As you say, this is a great idea in principle. In reality I think that it's really difficult to make it work.
I don't think its about best practices, its about good design and communication. Even if we are just talking about PhD students, the majority of them are fresh graduates. They are no different than fresh grads in a company. Those grads work with experienced senior software engineers to guide them and provide design advice (not just best practices). Those engineers are often the ones writing the complex/difficult areas of code.
> RSEs should not be writing code for students doing PhD level projects in my opinion
So should a mechanical engineer PhD be designing and making all their own robot parts? Or should the shop engineer help them? The few mechanical engineer PhD's in robotics I know made a few early prototype test parts themselves with help from the shop engineer, but the shop engineer made and even helped design most of it, especially the final prototype.
> As you say, this is a great idea in principle. In reality I think that it's really difficult to make it work.
The point I'm making is that it does work and its proven to work very well (which is why the major industry labs do it). In my experience its Academia that doesn't like it. Anything which appears to take power/freedom away from scientists and gets in the road of their research is rejected. Though I think the core reason is (as other comments have mentioned), there is no incentive for Academia to make it work. The funny thing is that having a RSE working with them would actually help the scientists in the long run and allow them to focus more on the research because they wouldn't have to do everything themselves.
> I don't think its about best practices, its about good design and communication.
I would argue these should be included in best practices for software engineering.
> So should a mechanical engineer PhD be designing and making all their own robot parts? Or should the shop engineer help them? The few mechanical engineer PhD's in robotics I know made a few early prototype test parts themselves with help from the shop engineer, but the shop engineer made and even helped design most of it, especially the final prototype.
This is an interesting example. Every mechanical engineer I know has huge respect for their in-house machine shops. Everyone has a story about some design they submitted for fabrication, only to be told by the machinist that their design was terrible and they should do it another way. Generally machining jobs are very well-defined though, you have to submit CAD documents, tolerances etc.
The shops in universities I've worked in have a strong incentive to help people optimise designs because they're the ones doing the manufacturing, and they know what sort of things will work and what won't. But by and large this is informal. Usually this comes in the form of "have you thought about designing this another way, because this is really difficult/expensive/time-consuming to machine". Maybe this is just a cultural thing for machinists?
The PhD question - if your project is to design a new type of part then you should probably do the design. Should you make it though? It depends if the project is specifically looking at fabrication. Otherwise it's normal to dispatch this to a workshop.
In my opinion, it comes down to what your PhD is training you for or what you're hired to do as a postdoc. If your job is data analysis, then I think you should be writing code, but you should be able to get guidance and support. If you're a field biologist with no coding experience and you want to develop an app to take measurements, then that's a case when contracting it out to an in-house development team makes sense. I'm not saying it can't work, but the make in making it work is important.
If you incentivize RSE's properly then their time will become expensive and we need ways of figuring out how to maximise their impact.
Well I think the best way would be for RSEs to maintain a project and PhDs should then contribute to that project via pull requests. RSEs can then point towards proper coding styles, test development, etc.
That would ensure that the contributions of the PhDs does not get lost and they learn how to properly contribute to a project.
See my sibling comment on this. I think this is an ideal case for an RSE: if you have a shared codebase that ends up being contributed to by multiple members. That avoids legacy problems where someone contributes, leaves, someone else modifies, none of it is in source control etc. However, this assumes that you have a group that is structured around some common IP or library - and sure, there are lots of places where this applies. This is generally more mature research, not something that a PhD student has just come up with.
There are of course scenarios where someone comes up with some very high impact work, and there's an obvious need to make it robust or user-friendly, spin-out, etc.
It works less well for groups where everyone works on different or loosely related projects. That's not an efficient use of an RSE's time, in my opinion. Though of course you can have a situation where lots of people do random projects using the lab's core code. In both cases, there is a use-case where RSEs embedded in a university can train students on good coding practices.
The challenge is that usually PhDs are not writing software that's designed for production, is very specific (for a single conference or journal paper) and often the utility of the code is nebulous until the end of the project. So what you don't want is for RSE's to spend ages writing code for a PhD project (which could be done by the student) only to have it thrown in the trash when the student leaves, or when they pivot to a new avenue of research.
I'm saying this as someone who did a PhD and who wrote a lot of code, including refactoring legacy codebases in my group.
Where the utility in having an RSE lies is where the group is all using some shared codebase that gets touched by everyone. This is the sort of cruft that I had to work with: legacy frankencode that generations of students and postdocs had added to. It would have made a ton of sense to pay someone to spend a year optimising it (which is ultimately what I did). But you want to make sure that RSE's maximise utility in the group. Having them work on individual student projects is not an effective use of their time IMO.
I'd say it depends a lot on the environment you'd be landing. I was hired to cleanup some clean room software and boy did it need cleanup. Simply by implementing some best practices and pretty much all what today goes as devops (this was 15 years ago in Switzerland) the thing got so speedy that a company got interested in buying it, so my contract was extended to add the final perks to close that sale. I refused the offer to join the buyer company and that was it for me. Ah and my name landed second on a paper, the only non-PhD on the list. All in all it was an exciting stunt while the pay was in line with what others said - 30% under the market average (they were even surprised why I wanted it).
Is there any solution to be found through analogy with established fields' relationship between engineering and research disciplines? How, for instance, do physicists share formal models which rocket engineers might apply? Certainly the physicists aren't out developing unit tests for engines which apply their theory, but nor are the engineers exasperated by physicists rejecting their standardized tools (I imagine?).
Well from experience the interface is conferences and papers. On conferences you get to know the latest stuff that is going on and the implementation details can be found in papers.
What needs to be said though is that reimplementing something from a paper is near impossible. I had to do that a couple of times and I was only fully successful if the paper was accompanied by some open source code as there always are tiny little edge cases or initialization details that won't be covered in the paper.
One could argue that bad software engineering practices are rewarded by academia. If no one can follow your work you drive away competition so why write comments. If you regularly refactor your work no one can use your API but you. Not unit testing ensures that code is cryptic, and people will have a hard time refuting your claims due to errors. The list goes on and on.
This last point, and replies to this all seem to talk about the research industry. But it has been my experience that this same dynamic of being “less than” exists in any industry that is not primarily software.
I am a mechanical engineer by degree, but a software engineer by 30 years of practice. That has allowed me to thrice straddle the divide where I worked on software that was not just software for softwares sake, but rather as a value add addendum/enablement to an institution whose roots were in more real-worldly devices. I also spent time on the other side of the divide working twice for companies whose primary product was software.
This “less than” is a very real thing. Software is still a very new thing relatively and it’s taken over the world in just 30 years time. The power dynamics, entrenched in generations haven’t had time to rebalance.
I’ve seen this run the other way as well. Where companies that are primarily software, look down on the other disciplines that participate, a necessary evil, but resented.
I have yet to find a company/institution where the trifecta of mechanical, electrical, and software is balanced in mutual respect. If one such exists and you are looking for someone that would love to work in that environment, drop me a line. I’m skeptical that there are any. It seems one discipline or the other always triumphs and tramples the others down. On off axis variant of Pournelle’s iron law of bureaucracy .
Just curious, have you sampled many robotics companies? I have worked with dozens and it seems like there are some with well-balanced power dynamics between mechanical-electrical-software.
I have to agree that in some environments you can see comp sci majors picking on anyone who isn’t part of traditional “software”.
I switched from engineering into comp sci at my university and I can safely say that the elitism for software was stoked by the elitism of engineering.
While I was in engineering, everyone bragged about how they were going to get “The iron ring” (a thing we give to only engineers in Canada). When I switched over to comp sci, all my old engineering buddies would say to me, “yeah, but you don’t get the ring” like it meant everything.
After I graduated, I noticed the comp sci “software” folks didn’t like engineering folks much (especially new grads). When I asked why, they shared similar stories to mine. The engineers and their pumped up pride at my local university had hurt their relation to other fields by being arrogant.
I’m not trying to justify this kind of prejudice, but the reasons it happens are fairly obvious to me. It’s sad cause all of us are so similar in our trades and will likely end up doing similar work as well.
I sent you an email too, but my employer REV Robotics definitely fits this description in my opinion, especially in regards to the software team's collaboration with the electrical team.
The last point is important. In my experience it's mostly about status. RSEs are always seen as inferior to researchers (research scientists) who supposedly come up with the "big ideas" while RSEs merely implement the stuff they're told to do.
In reality the line is much blurrier. There can be no innovation and iteration without implementation of ideas and the RSE work is just as important.
But unless this view changes, nobody wants to be an RSE.
According to a quick search for the case of researchers, and the stereotype of most staff in academia across the board being underpaid, this means academic researchers would have even less incentives/perks/reward, because now they have one less beneficial status differential.
Is there some way to make human respect feel like it's not a zero sum game? The world may never know.
> Is there some way to make human respect feel like it's not a zero sum game?
Respect is non zero sum but status is definitionally a positional good. If you’re number 1 someone else isn’t. There can be uncertainty about status but ambiguity always collapses eventually. Everyone can be treated with respect but there will be a prestige or dominance hierarchy in any group of humans, subtle as it may be.
I actually wouldn't mind being lower status than the scientists, so long as my particular expertise was respected and I had a reasonable degree of autonomy within my domain.
I did tech support at a university for a bit. I certainly wasn't as high status as the professors, but they mostly respected me and the value I provided (especially when you do things like help them recover that next book they were working on, or whatever).
Not just viewed, but literally called an assistant in the official title.
I am a licensed professional engineer (mechanical) and I work in academia (though my work is varied and does involve some code), my official title (in french) is "Professionnel de recherche", which translates to Research Professional. It is my understanding that in most of the english-speaking world, this position (someone that works for a research lab, who is qualified above the technician level, but not a PI nor a postdoc or student) is called "Research Assistant".
I write "Research Engineer" on my résumé/linkedin, because TBH for most people who don't have much experience in academia, "Research Assistant" sounds like an admin assistant (secretary).
I'd argue the position of software engineers/ programmers being those that implement the thinking of the business/science thinkers isn't unique to academia.
Apart from compensation and freedom you also get micro managed by someone who might know about the science part, but is really bad on the software architecture and engineering part.
> even if you have a PhD many will see you as "less than."
Inside academia and outside of it.
Inside academia you are not on a tenure track or similar, and will have to put up with a lot.
Outside academia, your peers will be making 3x or more than you, working less hours, with less stress, etc.
The reason RSE's jobs are hard to fill and often aren't even opened is that they don't make sense. If you are good enough for an RSE job, you will be good enough for research postions at FAANG. Those pay 10x more, so you also need someone willing to not accept that 10x pay, and also willing to work double the hours.
RSEs making a reasonable pay for the skills they require make no sense either, because that would put your pay at 2x that of professors, etc.
I'm curious which company pays a RS 10x more than a SE/RSE. I've found its usually SE/RSE's that make a bit more than RS's but I've never seen an RSE comp significantly outweigh an SE unless its in ML/Crypto.
On that note I wish levels.fyi had RS and RSE roles...
This isn't totally crazy but it's only if you stretch the facts enough. An RSE at Oxford will earn 32k GBP, like a postdoc. In theory, that person could be so good they could get the highest starting salary possible at a Tier 1 paying company like a research engineer at Hudson River Trading, which can be > 320k TC. So it's possible but only for a very very small number of people. 3x-5x is much more likely.
> highest starting salary possible at a Tier 1 paying company like a research engineer at Hudson River Trading, which can be > 320k TC.
You don't have to be at a "tier 1 paying company" to exceed 320k TC, that's easily achievable at any tech company that has gone public in the NYC metro or Bay Area. TC at a FAANG (MANGA or whatever they're calling it) for a more senior IC role can easily cross the $500k mark. If you've been their awhile and have been accruing shares that have increased dramatically in value, crossing the 7 figure mark is not unheard of.
So you don't have to stretch the facts too much to realistically achieve the 10x. If you take two very qualified engineer graduating with a PhD, one chooses to go to Google and stay there the other goes to a university and works as an RSE. Fast forward a decade, you'll definitely be seeing a 10x difference in total comp.
10x more than academia. I’m a PhD in mineral physics turned Big Four MLE via startups and I earn about 10x what I would be on if I had stayed on the academic track.
I certainly agree - it was about 10 years ago now, but I did a physics degree and ended up pivoting to computer science. I offered to help a couple of friends in the department with their analysis, and ended up writing the majority of their code in actually analysing their experimental results. In both cases, their supervisor decided not to put my name in the resulting papers, and treated me less than a lab tech.
Why would anyone want to do that, when a six-figure tech company salary is just next door?
While the academia has a lot of issues, software engineers should also adjust their expectations.
First, if you choose to work for a non-profit, you should expect a huge pay cut. That's the way the society works. Or actually it's the opposite. If you work in a for-profit role, the society allows your employer to keep most of the value you create. If you create a lot of value, that gives you the leverage to negotiate much higher compensation than your peers in less profitable roles.
Second, like in any business, you should focus on what brings the money in if you want to advance your career. You should be the PI instead of the lab technician. Instead of building the software other people ask, you should become an expert in the field and build the software other people will need in the future. For example, bioinformatics as a whole used to be more of a support role, but today there are many high-profile bioinformatics PIs.
Third, many people in the industry fail to realize that you don't really work for your employer in the academia. You work to build a reputation for yourself, which makes you valuable to the employer. If you want to continue working for other people, the upward career path from support roles goes to the administration.
>First, if you choose to work for a non-profit, you should expect a huge pay cut.
It may be the culture in some ecosystems but it's entirely false. A nonprofit can pay competitively with organizations of its size. I've worked at nonprofits and I've seen this line towed far too many times as an excuse to lower labor costs for people producing value for the organization. If your nonprofit is service oriented in anyway and requires people, appropriately investing in those people through their compensation and WLB is an appropriate investment.
In theory, a nonprofit should be able to outperform and outpay a for-profit entity of the same size, largely because there shouldn't be an expectation of the organization to cut a big slice out to investors but instead to invest back into itself. This could mean increased hiring, grabbing top talent, investing back in a cause and so on.
While some non-profits may be able to pay competitive wages, especially for people whose skills are not in high demand, that's definitely not true for most of them. In general, non-profits don't compete directly with for-profit businesses. They are far more likely to serve niches where for-profit businesses are not viable for one reason or another.
Very often, the level of funding is what it is and the non-profit has very little control over it. If you choose to pay higher salaries, you get less work done. This is common in the academia, which is usually funded by taxes and tuition fees.
Sometimes there is even an inverse correlation between funding and salaries. If a charity chooses to pay higher salaries, its "administrative costs" increase. Donations may then dry up, because people consider the charity inefficient.
Many non-profits rely extensively on volunteer labor. Salaried professionals often contribute their time and expertise to a worthy cause for free. That puts paid employees in an awkward position. It's hard to argue that you deserve a higher salary when the market rate for your services is 0.
Principal Investigator, the feudal lord who has near total power over the post docs and total power over the grad students in their lab. They can ruin your career at will and will almost certainly suffer no consequences whatsoever from doing so.
> Third, many people in the industry fail to realize that you don't really work for your employer in the academia. You work to build a reputation for yourself, which makes you valuable to the employer.
That feels like how it works outside of academia as well.
There is a huge difference: Universities don't do reseach, and they don't care about the research their employees do. If you switch jobs, your old employer doesn't hire anyone to continue your research. Instead, you usually take your project to your new job.
In the industry, your employer tends to own the projects you work on.
The RSE role doesn't seem to fit into their current model. It doesn't work from a career path, nor from a competitive compensation standpoint.
I suspect that the housing crisis is also going to push more people out of these positions.
If you have a PhD in material science, physics, chemistry, biology, etc... and are reasonably knowledgable in Python (or similar), perhaps spend a year as a postdoc.
After that year, seriously consider moving to a tech company.
your premise is that it's not being sold right but maybe you just need more people to give their perspective. I might be considered a research software engineer in the US. In my job, I get paid (well) to develop open source software for data visualization, and I work remotely. I get to maintain existing tools and develop new ones. there is an interesting global community that asks questions daily, and I am constantly learning. I'm not working under the thumb of overbearing people, just delivering results and contributing to new grant goals to continue the funding. these positions are not necessarily common, but are interesting and cool.
I'll offer another positive perspective. I've worked as a software engineer within the Department of Energy's National Laboratory System for 15 years, and I really enjoy it. Software is a major element of much of the laboratory's work, and in some cases such as mine software is the main product. We enjoy autonomy, lead projects as PIs, and develop mostly open source software. We are also hiring https://nrel.wd5.myworkdayjobs.com/NREL/2/refreshFacet/318c8....
Hi FlyingRobot --- I'd be interested in hearing more about your career path (and openings at NREL). What I've personally heard and experienced aligns with the overwhelmingly negative accounts elsewhere in the thread, which (I feel) have also correctly identified the misalignment of incentives that's responsible.
On the other hand, I've never worked in the national lab system. If you're still monitoring this thread, I'd appreciate it if you contacted me at the email in my profile!
I will second this. The national labs are definitely some of the places who know what to do with research software engineers and treat them right for the most part. The Computer Systems Engineers and Software Engineers I had the chance to meet at LBNL had a decent amount of autonomy and were very good.
If RSEs are actually valuable but under-compensated, can someone please disrupt this industry?
Would love to see more startup medicine development companies ...
Alternatively, would it make sense to provide RSE services as a company, for very high hourly rates, separately from the rest of the university, so status and rules are less of a problem?
The reason they're under-compensated is that they aren't valued, unfortunately. A lot of scientists are self taught programmers and don't know what they don't know. The other issue is that if you say you should spend half your departments budget on hiring much better paid software developers, it looks really bad:
a. You're implying your colleagues are incompetent.
b. You can't use that money for "science".
c. It puts in painful relief that the software guys are much better paid than you are, implying that their work is more valuable than yours. But a lot of scientists put up with low pay because they believe their work is really valuable.
Absolutely, but I will give them the hand that achieving a good software dev experience is extremely hard inside a uni. There is a number of ways that they need to match the industry and are having a tough time doing so due to the institutionalization of all the adjacent processes.
> From the perspective of the university and the PI, they can just get a graduate student to do the work for 1/5 the cost.
That is what they do. However, some projects grow in scale where a grad student cannot handle it well, or if they try, it will be a detriment to their research. To give you an example, only one of the PhD students (non-CS engineering) in my group and the ones around me had taken data structures or algorithms - yet everyone's thesis was "computational" (i.e. numerical computation, etc). None of the advisors would appreciate their students going off and taking serious CS courses.
When a research team wants to go to the next level, they need to hire someone with better SW skills.
From my experience "science" does not appreciate RSEs, the compensation is bad, the freedom is bad, and you get all of the big company bureaucracy from the University.
It's just a bad environment and to top it off, even if you have a PhD many will see you as "less than."