Or at any rate into the chassis; all those PCIe lanes have to go to devices that are mounted somewhere.

We have disk shelves now, wouldn't the next step be a "network shelf"? And probably a "GPU compute" shelf later on with some kind of high-speed high bandwidth backbone tying them all together. Turn the whole rack into a glorified backplane. ¯\_(ツ)_/¯ sounds like something IBM would do.