It's a very awkwardly phrased recursion problem. The point is to find a unique x, y where x <= y. The algorithm starts with the set of all possible (x, y) pairs, and switches between the Bob and Alice stages.
In Bob's stage, eliminate pairs that give a unique value for x^2 + y^2. If such a unique value matches the value Bob was given, then obviously you know x, y for sure.
Otherwise, switch to Alice's stage, which operates on the remaining pairs from Bob's stage, and eliminates all pairs that give a unique value for x + y.
The weirdness in the question is both Alice and Bob are performing the exact same algorithm, but (to use CSP parlance) there is a channel from Bob to Alice in Bob's stage, and vice versa, due to the different information given to both parties. Alice must wait for Bob to declare if the algorithm has terminated or not.
I don't think the algorithm you're proposing works, although that was my first idea as well. And I don't a priori see why the pair (x,y) would have to be unique (but it does appear to be, even if instead of choosing x and y in 0..5 you choose from all natural numbers).
In Bob's stage, eliminate pairs that give a unique value for x^2 + y^2. If such a unique value matches the value Bob was given, then obviously you know x, y for sure. Otherwise, switch to Alice's stage, which operates on the remaining pairs from Bob's stage, and eliminates all pairs that give a unique value for x + y.
The weirdness in the question is both Alice and Bob are performing the exact same algorithm, but (to use CSP parlance) there is a channel from Bob to Alice in Bob's stage, and vice versa, due to the different information given to both parties. Alice must wait for Bob to declare if the algorithm has terminated or not.