Or the people that rely on the tools to make decisions...

https://sheetcast.com/articles/ten-memorable-excel-disasters

Yes, and it's not as absurd as it might seem:

Imagine hooking up all ICBMs to launch whenever this week's Powerball draw consists exclusively of prime numbers: Absurd, and nobody would do it.

Now imagine hooking them up to the output of a "complex AI trained on various scenarios and linked to intelligence sources including public news and social media sentiment" instead – in order to create a credible second-strike/dead hand capability or whatnot.

I'm pretty sure the latter doesn't sound as absurd as the former to quite a few people...

A system doesn't need to be "true AI" to be existentially dangerous to humanity.

How is a calculator going to cause harm? Assuming you get an industrially rated circuit board when appropriate, it should work just fine as a PLC.

If you try to make it drive a car, I wouldn't call that a problem of giving it too much power.

I'd say by far our biggest problem for the foreseeable future is granting other humans too much power.