I like ceph in some cases, but ceph hasn't worked very well IME with multi-region clusters or just higher-than-lan-speed latency.

I never tried the actual multi-site support, but for my own homelab use-case - I don't have enough servers in each region to make their own ceph cluster.

I don’t think any strongly consistent posix data store will work well in a multi region setup.

Each write needs to be propagated before the application is unblocked. When you have tens of milliseconds of site to site latency this means every IO request is going to have worse performance than spinning rust.

Object storage can work better in this regard because the IO chunks tend to be larger and therefore the round trip time is traversed less per bit written.

Another alternative is that you run your regions in active passive setup so there is only one active region at a time and when the active region is changed all in process state in the previous active region can be sync to the new active region. Writes only need to be synchronously written within the active region.

Honestly, if you can do that you can make the big bucks at FAANG.

I personally just run a single home NAS which backs up to S3. Very strong durability in that S3 is never going to lose my data at the same time my NAS goes bad, but not great availability in that I will need to rebuild the NAS and pull down files from S3. When it comes to my home tech strong durability is a must but availability, not much so.