Will an AGI have the ability to decide for itself? If so, then how can you make it what you want it to be (with any certainty)? And if not, then how is it "general"?
To me, it's kind of like raising kids. You try to train their neural nets to bias them toward doing what you think is good and right. And that sometimes works. Yes, I think it's fair to think of it as biasing the dice. But it's sure not 100%. They'll still decide which of your values they keep, and which ones they throw away as being stupid. And you can't stop them from doing that.
I guess, to try to respond to your direct point, that if it's an AGI, then it's less deterministically driven by the training data than we might wish.
To me, it's kind of like raising kids. You try to train their neural nets to bias them toward doing what you think is good and right. And that sometimes works. Yes, I think it's fair to think of it as biasing the dice. But it's sure not 100%. They'll still decide which of your values they keep, and which ones they throw away as being stupid. And you can't stop them from doing that.
I guess, to try to respond to your direct point, that if it's an AGI, then it's less deterministically driven by the training data than we might wish.