Anyone who has done a lot of bit twiddling in C has other uses for ^. (I find it ironic that 4^5 has an obvious meaning outside of programming that it doesn't have in a large variety of widely used programming languages.)
I suppose they should be allocated based on frequency of use within the intended domain of the language, with the most frequent getting first choice. I guess pointer dereferencing would be more important than xor for most domains. Not that I am particularly advocating ^ for pointers, * is just as good, but I do think using @ for the opposite of its meaning in Pascal would be confusing.