If data is being transferred frequently between RAM, the CPU, or the GPU, it will kill performance anyways. So maybe a GPU is actually one of the better things to have over a socket..
If your OS is able to deal transparently with heterogeneous hosts across different network transports, then the GPU and its memory will end up being treated as just another compute node hooked up to one of the other compute nodes by a ridiculously fast network.
