It's repeating the same patterns it learned from text. Maybe with different specific objects, but still. One good way to test this is to ask it to count. It will break very soon. A person is able to build a rule in their head: "one apple is 1, two apples are 2, three apples are... 1+2 = 3".
Prompt: "One apple is 1, two apples are 2, three apples are"
Model output: "One apple is 1, two apples are 2, three apples are 4, four apples are 5, six apples are 7, seven apples are 8, nine apples are 10, ten apples are 11 (for apples being the perfect length of life)."
Even if you use a special dataset to each it to count, it won't be able to count beyound the examples in the training set. So it's spewing plausible-sounding gibberish at you (i.e. approximating the training set distribution)
It doesn't generalize. Not in the sense that it can't give you a phrase that didn't exist in the training set. It can. But it can't give you a new kind of phrase, of a "kind" that didn't exist in the training set.
>But it can't give you a new kind of phrase, of a "kind" that didn't exist in the training set.
Can you give a concrete example that doesn't involve math (which GPT is a bit handicapped at, because of the way its encoded)? I feel this is a bit like 'moving the goalpost'. It seems to me plenty of humans only ever repeat things they've heard and aren't coming up with novel, complex abstractions or ideas...
Let's test GPT-2: https://huggingface.co/gpt2?text=One+apple+is+1%2C+two+apple...
Prompt: "One apple is 1, two apples are 2, three apples are" Model output: "One apple is 1, two apples are 2, three apples are 4, four apples are 5, six apples are 7, seven apples are 8, nine apples are 10, ten apples are 11 (for apples being the perfect length of life)."
Even if you use a special dataset to each it to count, it won't be able to count beyound the examples in the training set. So it's spewing plausible-sounding gibberish at you (i.e. approximating the training set distribution)
It doesn't generalize. Not in the sense that it can't give you a phrase that didn't exist in the training set. It can. But it can't give you a new kind of phrase, of a "kind" that didn't exist in the training set.