This is not intrinsically true. Cooling and power delivery certain have it beat, but it seems feasible to beat based on memory bandwidth. I bet a GPU uses roughly the same order of power/cooling that say half a threadripper does?

No, how do you beat the wide bus memory bandwidth of a discrete GPU with the standard dram bandwidth? Even if your processor was not competing with the GPU for bandwidth you'd still have a small fraction of the memory bandwidth available.

You'd have to come up with a new architecture to make this work, but perhaps you could have CPU/GPU on the same chip with dedicated GDDR memory lanes.

PS4/XBOne have an APU with powerful (ish) graphics - how does their memory work?

They work with graphics memory rather than regular memory if my memory serves. Hehe.

Aw, I was thinking bw between the CPU and GPU directly, my bad.

Aren't console GPU chips on the same die as the CPU ?

Might be more accurate to say that consoles put the CPU on the same die as the GPU. The GPU portion of the console SoC die is bigger than the CPU portion and the memory controller is very much a GPU memory controller not a CPU memory controller.