The AXI bus internal to the Pi's SoC is only capable of about 4gbps, and it carries DMA, so ~2gbps is more or less the hard limit for any kind of combined IO operation like disk<=>network no matter what kind of hardware you use for disk and network.

So yes, each pi can easily saturate its own 1gbps interface, so a system like ceph that parallelizes reads and writes among nodes is severely crippled by the onboard switch choking off bandwidth to external clients. For the same reason, you can't easily scale this platform beyond a single board, which puts your clustered system back into a single point of failure.

Strange, but terrible design decision.

Pi raw reads off ssd are higher so yes there would be a benefit. Reckon it could saturate 2.5.

The internal AXI bus on the Pi SoC can only move data at ~4gbps by DMA, so no the Pi 4 simply cannot cannot round trip real data from disk to network any faster than about 2gbps. It can saturate a 2.5gbps nic with a synthetic datastream, but for a storage application the 1gbps interface is as much as a single pi is going to realistically be capable of