I just setup a test cluster at work to test this for you:
4 nodes, each node with 2x SAS SSDs, dual 25Gb NICs (one for front-end, one for back-end replication). The test pool is 3x replicated with Snappy compression enabled.
On a separate client (also with 25Gb) I mounded an RBD image with krbd and ran FIO:
4 nodes, each node with 2x SAS SSDs, dual 25Gb NICs (one for front-end, one for back-end replication). The test pool is 3x replicated with Snappy compression enabled.
On a separate client (also with 25Gb) I mounded an RBD image with krbd and ran FIO:
I get a consistent 1.4 GiB/s: