As a (non-PhD) researcher who works for one of the last traditional industrial research labs in Silicon Valley, I've noticed this trend for years. I've seen industrial research labs either close or become increasingly focused on short-term engineering goals. I also have noticed that newer Silicon Valley companies have adopted what Google calls a "hybrid approach" (https://research.google/pubs/pub38149/) where there are no divisions between research groups and product groups, and where researchers are expected to write production-quality code. I've noticed many of my PhD-holding peers taking software engineering positions when they finish their PhD programs, and I also noticed more people who were formerly employed as researchers at places like IBM Almaden and HP Labs switch to software engineering positions at FAANG or unicorn companies.
Unfortunately, as someone who is also working toward finishing a PhD, I've seen very little guides for CS PhD students that reflect this reality. To be honest, I love research and I'd love to stay a researcher throughout my career, but I don't have the same love for software engineering, though I am comfortable coding. Unfortunately with these trends, traditional research is now largely confined to academia, which is very competitive to enter and where COVID-19's effects on its future are uncertain at this time, and federal laboratories such as Los Alamos and Lawrence Livermore. I fear losing my job (nothing lasts forever) and having to grind LeetCode since there are few other industrial research labs, though given the reality maybe I should start grinding LeetCode anyway.
When I was doing my PhD (CompSci) in the late 2000s I looked forward to join Microsoft Research because it seemed they were doing really cool stuff, even at a time when "corporate" Microsoft was the bad kid (me being very pro-Linux at the time).
One thing lead to another and I ended up going to another Research Assistant role and then getting fed up with research and going into industry.
But the idea of the Xero lab, Sun Microssystem and later Microsoft research was something I always really dreamt about. Nowadays I think startups doing autonomus vehicle technology is what looks similarly disrupting to me, but I am too much down into the SaaS rabbit hole.
MS Research does cool stuff but MS is comically bad at turning the results into successful commercial products. By all rights, in terms of research results and focus, the iPad should have been a Microsoft product. Microsoft remains good at making faster horses...
You only mentioned the DOE national labs in passing-I encourage you to give them more thought. I’m currently a researcher at one of them, and if you land in the right group/application area it can be truly meaningful work.
If you’re in the bay I encourage a closer look at LBNL and LLNL, if you’re open to moving, NREL is in a great location too. Obviously Sandia, LANL, ORNL, and PNNL all do good applied CS work as well but the locations are a little more remote and options are better if you can get/hold a clearance.
I know a (non-PhD) researcher who moved on after one of those labs closed and sadly he finds it impossible to get a research job again. Academia is his only option but he'll have to earn a PhD before getting such a job...
It’s a tough situation, especially for non-PhD holders. Even with a PhD it’s possible for a researcher to enter a rough patch where a career change looks inevitable. The last time I was unemployed I started sending out my résumé to various community colleges and universities to enter their part-time/adjunct lecturer pools (I have a masters), but unfortunately my lay-off period didn’t match well with the academic hiring period. I managed to get a few interviews for software engineering positions, but the algorithms-based interviewing bar is very high these days for a software engineering job, and admittedly my software engineering chops degraded due to spending years in research. Ultimately I ended up finding an AI residency program at a research lab, which is how I ended up at my current position, but it required switching research areas and taking a minor salary drop.
"Job" or "research job"? I mean, yeah, it's tough to find a job purely navel-gazing without a PhD (if then), but we hired all the refugees from PARC, Digital Equipment WRL and Olivetti Research Center we could. Most of them made the transition to a more product-focused, results-measured workplace. Maybe adjusting expectations would help.
You should start grinding LeetCode if you plan on interviewing for industry jobs, because it might help you survive those awful white board/algorithm puzzle interviews.
As a rule of thumb I would say that building something "production-quality" takes at least ten times longer than building something I and close collaborators can use. By having researchers build production-quality code the spend a lot more time with software engineering and a lot less time researching. (And of course many great researchers are actually not that great at programming, so you probably lose many of them as well)
Because the skill sets aren't the same. Research in of itself is a different skill set. Getting a PhD is more about learning how to take abstract and vague ideas and turning them into a reality. Making highly readable, robust, and secure code is a different skill set. Sure, you can have people that do both, but at that point you're often asking a painter to do sculpting. Skills do translate, but not well. They probably won't be interested in it either, so you're not using your workers efficiently either. Give the canvases to the people who love painting and give the marble to those that love sculpting. You'll probably get better art in the end.
Unfortunately, as someone who is also working toward finishing a PhD, I've seen very little guides for CS PhD students that reflect this reality. To be honest, I love research and I'd love to stay a researcher throughout my career, but I don't have the same love for software engineering, though I am comfortable coding. Unfortunately with these trends, traditional research is now largely confined to academia, which is very competitive to enter and where COVID-19's effects on its future are uncertain at this time, and federal laboratories such as Los Alamos and Lawrence Livermore. I fear losing my job (nothing lasts forever) and having to grind LeetCode since there are few other industrial research labs, though given the reality maybe I should start grinding LeetCode anyway.