The fact that our low-power chips also happen to be lower die size is an artifact of path dependence. The primary market for lower power was battery powered devices of which phones are by far the most numerous. So, the lower power chips started there and didn't have much to do. As those have gained sophistication with time, they have also grown in die size.
Xeon and server chips generally want to maximize memory bandwidth--and they make a whole series of architectural tradeoffs to accommodate that.
Phone chips generally want to maximize power efficiency and basically don't care about memory bandwidth at all. They effectively don't want to turn on the system memory or flash, period, if they can avoid it. One way to do that is to cache things completely in local on-chip RAM.
Computer architects will make completely different tradeoffs for the two domains.
I suspect that Apple wants way bigger caches. And chip size (especially RAM) directly impacts yield and cost.
Apple is probably more than happy to double the cost of the CPU if the extra cache gets them 20% more battery life.
Nobody in the commodity CPU market would ever make that trade.