>So it is with any technological innovation. Should computers not have been invented because they eliminated jobs? Should steel? What about agriculture? The future is sure to be different, but that doesn't mean we should fight to deny progress. That way lies the Luddite and Conservative. It's only possible to use new tools for good, not try to erase them to prevent evil.

It's not even necessarily evil, right? It's more likely just one big "whoops, majority of us didn't see that coming, and/or didn't agree with those who said it was". But that whoops lands you in a position that you may not be able to recover from.

Ben Geortzal makes your exact argument here: https://youtu.be/MVWzwIg4Adw?t=2795 and in the same interview he is talking about how his company already spent time and money on building decentralized infrastructure for it to run on "so that no one person can turn it off and stop the singularity" and gives his take that the takeoff is likely to be on the order of years and that it's going to be very rough.

So, he at least seems to acknowledge that it's very risky, and that we have no idea what we're actually walking into, but he also just doesn't seem to care if that happens to be a one-way door we walk through and find a big "oopsie" on the other side of.

This is the guy credited with popularizing the term AGI in the first place. I dunno man... am I wrong to be worried?

So it is with any technological innovation. Should computers not have been invented because they eliminated jobs? Should steel? What about agriculture? The future is sure to be different, but that doesn't mean we should fight to deny progress. That way lies the Luddite and Conservative. It's only possible to use new tools for good, not try to erase them to prevent evil.

I think that many technologies, ones that we continued using, are good. The ones that turned out to be bad, we banned. If you make a list of technologies we are still using, then it will contain good ones.

I think that actually, we would be better off as humans if we could figure out a way to coordinate (not easy) and decide in advance which technologies we allow to be released into the world. It wouldn't be perfect, as we might still make mistakes, but we maybe could have stopped leaded gasoline, CFCs, Thalidomide, social media feeds, and so on.

If it's a big enough evil, yes, sometimes we should not invent some things. And I say this knowing that there are currently drugs that the FDA is holding back (due to over-caution) even though they would be very likely to save lives. Sometimes we don't take enough risks. Sometimes we take way too much.

I don't know if stopping unaligned AGI is possible, but I think it's worth trying. I can imagine some good coming from aligned AGI, but I feel like most of the things we could do with aligned AGI, we could also do with just regular old narrow AI, but slower. Slow sucks when people are dying, but if there is the possibility of EVERYONE dying and nobody new being born ever again, then that's the thing to avoid.