It got the value of the sixth and seventeenth bits, moved them into the right positions, and inserted them into the original value. Off a one-line description written in English! I really cannot empathize with the idea that this is not a meaningful capability. If intelligence only means to you “equal in all capabilities to an experienced human”, you are never going to be able to see anything coming ever.
If you ask CoPilot to solve something it hasn't seen, it won't be able to solve it.
It's a transformer. Do you understand what that means? It's just matrix multiplication.
It generates maximum likelihood token strings, based on its training data.
It doesn't "understand" what those token string mean.
You are amazed because you're testing the transformer by asking the transformer to generate human-written code THAT IT WAS TRAINED ON. To make CoPilot fail, all you have to do is ask it to generate something unlikely, something it hasn't seen in training.
