The fact that our low-power chips also happen to be lower die size is an artifact of path dependence. The primary market for lower power was battery powered devices of which phones are by far the most numerous. So, the lower power chips started there and didn't have much to do. As those have gained sophistication with time, they have also grown in die size.
Xeon and server chips generally want to maximize memory bandwidth--and they make a whole series of architectural tradeoffs to accommodate that.
Phone chips generally want to maximize power efficiency and basically don't care about memory bandwidth at all. They effectively don't want to turn on the system memory or flash, period, if they can avoid it. One way to do that is to cache things completely in local on-chip RAM.
Computer architects will make completely different tradeoffs for the two domains.
But who wants an expensive low power chip? Only those who make phones. aka Samsung and Apple.