In-built is the assumption that one person must have both competencies. This may be true for cash-strapped startups but this hardly plays well as general advice.
Definitely not ... I am a SWE in a big company and working on ML projects for the past 4 years. We have a data scientist team which does the data exploration and comes up with a model. Their output would be a jupyter notebook.
Then we have a team of "Applied ML practitioners", which I am a part of, we productionize the jupyter notebook, by setting up pipelines, services etc. We understand ML algos, stats, probability etc, but not as much as our data scientist team does.
Having both in the same person would be good, but is not necessary.
Regardless of how low you get your communication overhead, it still exists. It's rare to find people who can both run and test all the infrastructure and model code, and notice that the transformation you apply on line 34876 of file foobar_now_with_added_ml.py is statistically inappropriate for your problem.
That's not even to mention the really hard part, selecting a good outcome variable and appropriate ways to measure the performance of your system once it hits prod.
You can definitely split this stuff between people, but it gets super-linearly harder as you add more people, so it's really incredible to find people who can do both (and honestly, there aren't that many of them (I'd like to say us, but I'm probably not there yet)).
> It's rare to find people who can both run and test all the infrastructure and model code
It's also unnecessary to do so as long as your institutional processes are capable of synthesizing multiple peoples' competencies across multiple disciplines.
How do you think any machine more complicated than a train car was designed? Do my mechanical engineers need to understand the intricacies of avionics?
> it's really incredible to find people who can do both
Absolutely, and I think you'd have a hard time finding someone who disagrees. But you made a very strong assertion about a "need" which requires much stronger arguments to support.
> It's also unnecessary to do so as long as your institutional processes are capable of synthesizing multiple peoples' competencies across multiple disciplines.
You clearly work at much better run companies than I do ;)
I sincerely wish you luck! Yeah terrible managers are awful. The feedback loop of "my boss won't do shit to help me so I'll figure it out on myself" to "I'm competent in every single technology here and could run the show myself with enough time" to "management recognizes my skills and puts everything on my plate even though I have no time" is pretty dangerous.
