I think this is a common problem and comes because we stressed how these models are not interpretable. It is kinda like talking about Schrödinger's cat. With a game of telephone people think the cat is both alive and dead and not that our models can't predict definite outcomes, only probabilities. Similarly with ML people do not understand that "not interpretable" doesn't mean we can't know anything about the model's decision making, but that we can't know everything that the model is choosing to do. Worse though, I think a lot of ML folks themselves don't know a lot of stats and signal processing. They just aren't things that aren't taught in undergrad and frequently not in grad.
> difficult to distinguish causation vs correlation
I mean this is an extremely difficult thing to disentangle in the first place. It is very common for people in one breath to recite that correlation does not equate to causation and then in the next breath propose causation. Cliches are cliches because people keep making the error. People really need to understand that developing causal graphs is really difficult, and that there's almost always more than one causal factor (a big sticking point for politics and the politicization of science, to me, is that people think there are one and only one causal factor).
Developing causal models is fucking hard. But there is work in that area in ML. It just isn't as "sexy" because they aren't as good. The barrier to entry is A LOT higher than other type of learning, so this prevents a lot of people from pursuing this area. But still, it is an necessary condition if we're ever going to develop AGI. It's probably better to judge how close we are to AGI with causal learning than it is for something like Dall-E. But most people aren't aware of this because they aren't in the weeds.
I should also mention that causal learning doesn't necessitate that we can understand the causal relationships within our model, just the data. So our model wouldn't be interpretable although it could interpret the data and form causal DAGs.
>It is kinda like talking about Schrödinger's cat. With a game of telephone people think the cat is both alive and dead and not that our models can't predict definite outcomes, only probabilities.
It isn't just our models that can't explain it, there are real physical limits which mean that _no_ model can predict what state the cat is in.
The only reason why cats are a more outrageous example than electrons is that we see cats behave classically all the time.
The only vaguely plausible explanation why cat states are impossible in general is that large quantum system become spontaneously self decoherent at large enough numbers of particles.
Yes, _no model can_ is an important part. But what I was focusing on is that the cat is either alive or dead in the box and not both. Just because we can't tell doesn't mean that's not true. Particles are observers and the wave function is collapsed from the perspective of the cat, but not from our perspective where we can't measure. But people misunderstand this as the cat behaving in a quantum state, which isn't happening. People have also assumed "observer" means "human", when particles themselves are observers. Which is why the cat is actually in a classical state (either alive or dead, not both), because within that box the particles are interacting. The confusion comes because the analogy being misinterpreted (the analogy assumes a lot of things that can't actually happen because it is... an analogy and to understand it you really need to have a lot of other base knowledge to understand the assumptions being made).
> _no_ model can predict what state the cat is in.
Perhaps you meant to say "...state the cat will be in when observed"?
Otherwise, an important nitpick applies: superposition means that the system is not in any single state, so there's nothing to "predict" - it's a superposition of all possible states.
Prediction comes in when one asks what state will be observed when a measurement is made. As far as we know, that can only be answered probabilistically. So no model can specifically predict the outcome of a measurement, when multiple outcomes are possible.
I mean even without being observed by humans, the cat is in either the alive or dead state. It can't be in both. It is just that our mathematical models can't tell us with complete certainty which state that is. (People also seem to think humans are the only observers. Particles are observers too)
I think what is not mentioned nearly enough is the need for isolation to prevent decoherence of the cat. You need to make sure the box is it's own universe totally disconnected from the rest. Then I'd say it is a little more intuitive that parallel insides of the box might exist in superposition.
But it also can't be a real cat. Because if it was a real cat then the cat itself collapses the the wave function. Literally any particle interaction does. Really what is important here is that us being on the outside and in a different reference frame (we're assuming we can't do any measurements of things inside the box. Think information barrier) we can't obtain any definite prediction of the cat's state, only probabilistic. The information barrier is the important part here.
