I feel like these are all reasonably obvious gigantic landmarks taught to anyone with a stats or pattern recognition background. It's sort of just the lay of the land with statistics, modeling, optimization.
That doesn't mean at all that more people shouldn't be aware of them and think critically about them. It also doesn't mean at all that the implications of these features of ML are understood by the entirety of the organization which is affected by them.
But I sort of feel like it's the (perhaps unstated) primary job of a data scientist/modeler/mathematician/statistician/what-have-you employed at any company investing in this kind of technology. It feels about akin to a company building their software without using version control. Sure some people do that, but... c'mon!
If you're failing to do it then you're probably not listening to your experts enough. If they're not screaming it's because they're wrapping blankets around their head trying to block out the noise.
After having skimmed the paper, I think it has a subtly different focus which is vastly more important to consider. In particular:
> At a system-level, a machine learning model may subtly erode abstraction boundaries. It may be tempting to re-use input signals in ways that create unintended tight coupling of otherwise disjoint systems. Machine learning packages may often be treated as black boxes, resulting in large masses of “glue code” or calibration layers that can lock in assumptions. Changes in the external world may make models or input signals change behavior in unintended ways, ratcheting up maintenance cost and the burden of any debt. Even monitoring that the system as a whole is operating as intended may be difficult without careful design.
In other words, due to either unavoidable or marketing-based complexities, ML projects have a tendency to blow up abstraction boundaries and induce loops leading to unexpected tight coupling. In other words, you can interpret this less as a feature of ML as a task and more as a feature of how ML algorithms tend to operate, especially when operated in "black box" mode.
ML may, when practiced in some ways, be antithetical to good software design.
Again, quoting the paper
> Indeed, arguably the most important reason for using a machine learning system is precisely that the desired behavior cannot be effectively implemented in software logic without dependency on external data. (Emphasis mine)
This rings very true in my experience building ML systems. It's an unexpected challenge in software terms. You're often used to many kinds of behavior as being undefined until runtime (e.g. needing user input before they can be understood), but typically the function of a program is something you fully internalize statically—this is the cornerstone of understandable code.
OTOH, ML says that even the very behavior of the system must be deferred until "runtime" information can propagate back. This is especially pronounced in online decision-making tools, but plays out over much longer time periods even with batch algorithm design. This "phase distinction" problem is something you will feel immediately in building ML-based code.
It's probably the case that black box ML algorithms exacerbate this. I don't have a lot of experience there, though.
That doesn't mean at all that more people shouldn't be aware of them and think critically about them. It also doesn't mean at all that the implications of these features of ML are understood by the entirety of the organization which is affected by them.
But I sort of feel like it's the (perhaps unstated) primary job of a data scientist/modeler/mathematician/statistician/what-have-you employed at any company investing in this kind of technology. It feels about akin to a company building their software without using version control. Sure some people do that, but... c'mon!
If you're failing to do it then you're probably not listening to your experts enough. If they're not screaming it's because they're wrapping blankets around their head trying to block out the noise.