I am familiar with this situation and agree it is a problem. But this is a deterministic, well known, algorithm rather than AI, isn't it? I would also say that this falls under bias rather than misalignment. I think bias is a much, much more realistic problem area than misalignment.

This is exactly what I'm talking about in the other response to your previous comment. The joke goes "AI ceases to be AI when we start to understand how it works".

But it doesn't matter if you call the software that kills us all "conscious AI that was misaligned" or if it was "deterministic code that had a bug" - the underlying problem that we're all dead remains the same. We're just arguing about labels instead of issues.

The AI that kills us all (if this happens) will very likely be a simple, deterministic algorithm that ran on A LOT of data.