I think it is because linear regression is not science fiction enough even if that is the best that can be done on the data. We didn't hire this AI guy to do simple regressions!
From experience, there are lots of companies looking precisely for an "AI person" who is going to use a bunch of complicated sounding buzzwords and build some overly complex thing without evaluating simpler options. This kind of person gets way more attention than someone who suggests that what the company wants to do can probably be accomplished with a simple shell script, or someone that tells them AI is not magic and in the absence of some data capturing a relationship it can't really do anything. It's a hyped up field and has attracted lots of people who care about hype. (I work in the field, not trying to denigrate it, my point is that many / most business adopters want "AI" for all the wrong reasons
Ran into this very thing writing software for customer support teams. There's a Data/ML team within the company and there was a lot of talk about getting them to help plan the software to add abilities to "learn" what customers wanted with their support requests. The reality of the situation is that you can just take a step back and look at the pain points for the CSRs, the highest cost calls and where the time is spent servicing the customer. In the end, it's pretty clear that we can just do a few queries up front to realize that "oh, this person ordered 2 items of the same size in their last order and haven't returned one yet, it's very likely that they want to return one" or any number of other common scenarios that were found and then prioritized based on a simple cost/benefit analysis.
Could the Data/ML team have done something cool and likely complex to show clearer patterns and maybe get something in place to continually refine that data? Probably. Would it have saved more money for the company eventually?
Unknown. Would it have cost substantially more money to design and implement? Pretty sure.
