> What context does an AI have for knowing what it even means to advance the game?
Whatever context we decide that it has. We can guide it to prefer scenarios where the character doesn't die, or go as deep as having it rate its own results with a weighted scale.
> Imagine trying to teach a jellyfish how to play Skyrim. That's kind of where we're at right now.
The difference is that we don't have the ability to both redefine the jellyfish's goals and modify how it controls its interactions with the world.
Whatever context we decide that it has. We can guide it to prefer scenarios where the character doesn't die, or go as deep as having it rate its own results with a weighted scale.
Then we're not really talking about AGI anymore, but a Skyrim-playing engine. Do you see the difference? A human being didn't evolve to play Skyrim, so building your AI with specific knowledge of Skyrim is cheating.
A proper AI should be able to learn to play Skyrim just as well as it could learn to play Chess or Super Mario Bros or any other video game we throw at it, including games not invented until after the AI is written. This is much, much harder than building something that can just play Skyrim.
Whatever context we decide that it has. We can guide it to prefer scenarios where the character doesn't die, or go as deep as having it rate its own results with a weighted scale.
> Imagine trying to teach a jellyfish how to play Skyrim. That's kind of where we're at right now.
The difference is that we don't have the ability to both redefine the jellyfish's goals and modify how it controls its interactions with the world.