It's hard to talk about this and clearly convey all meaning.
If you're saying that it will decide because neural nets are black boxes which we don't have a complete understanding of, and we're without a clear way to analyze their behavior, I can see where you're coming from.
But these things will not be beyond our influence. They're going to be slaves to the computations encoded in the neural net connections / weights. We're going to shape / mold them through a process akin to natural selection. We're going to select for intelligences that want to help humanity. It's not going to be a roll of a normal dice, it'll be more like the roll of a weighted dice. And we're going to, I believe, get better tools / theories for understanding the output of these neural nets so we will be able to conduct this selection with some confidence.
Humans can be said to decide what we're going to do thanks to an uncaring, unconscious, and brutal evolutionary process that prioritized self-interest, survival, and reproduction. It's all about the selection process, and this time around we have a hand in guiding it.
Will an AGI have the ability to decide for itself? If so, then how can you make it what you want it to be (with any certainty)? And if not, then how is it "general"?
To me, it's kind of like raising kids. You try to train their neural nets to bias them toward doing what you think is good and right. And that sometimes works. Yes, I think it's fair to think of it as biasing the dice. But it's sure not 100%. They'll still decide which of your values they keep, and which ones they throw away as being stupid. And you can't stop them from doing that.
I guess, to try to respond to your direct point, that if it's an AGI, then it's less deterministically driven by the training data than we might wish.
If you're saying that it will decide because neural nets are black boxes which we don't have a complete understanding of, and we're without a clear way to analyze their behavior, I can see where you're coming from.
But these things will not be beyond our influence. They're going to be slaves to the computations encoded in the neural net connections / weights. We're going to shape / mold them through a process akin to natural selection. We're going to select for intelligences that want to help humanity. It's not going to be a roll of a normal dice, it'll be more like the roll of a weighted dice. And we're going to, I believe, get better tools / theories for understanding the output of these neural nets so we will be able to conduct this selection with some confidence.
Humans can be said to decide what we're going to do thanks to an uncaring, unconscious, and brutal evolutionary process that prioritized self-interest, survival, and reproduction. It's all about the selection process, and this time around we have a hand in guiding it.
Just my two cents anyway.