What's a bit worrying here is that life has had billions of years of evolution to shape its instincts (i.e: empathy, self-preservation, desire to explore, etc.). Our basic instinctual drives are tuned in a way that the system is more or less stable, the species doesn't go extinct.
We're in the process of creating machines whose drives will be completely artificial, not shaped by natural selection. We're going to shape them... Or, more accurately, greedy corporations, billionaires and leftists are going to imprint an approximation of their own morals and goals in there, for better or worse. Those machines will also inevitably end up a lot smarter and more capable than we are. It's possible that if we fuck up the programming, we just just won't be able to stop them from transforming the world into whatever they want... If the world they want is toxic or unsustainable, good luck trying to reprogram a machine that's faster, stronger and smarter than you. It's going to be the other way around, the machines will reprogram us if it suits their needs.
Wait, that's a weird list: "greedy corporations, billionaires and leftists" will push their own views and morals? Why do you conflate those things? In the US, leftists might want crazy things like universal health care. I do agree there is huge danger in programming systems and letting them make choices. But we already let it happen in things like letting software trade stocks and move the market (triggered by signals, causing unexpected behavior).
You only need to add "You are a nice AI." to the prompt to make it nice. Or you can replace "nice" with whatever you want, but then you suffer the consequences.
