2) never use distributed Filesystems unless you need a global namespace
3) distributed filesystems mean that you have a single point of failure, its just more complex and subtle form of failure.
4) You have to think about how partition works, and how you want to recover.
Sharding standard NFS fileservers is by far the easiest way to scale, have redundancy, and partition your failure zones. using autofs on a custom directory you can mount servers ondemand. /mnt/0 /mnt/1 /mnt/2 etc etc etc.
You then have the ability to chose your sharding scheme for speed, redundancy or some other constraint.
Nowadays the performance that distributed systems allowed are commodity. A single 4u storage server can saturate a 100gig network link with random disk IO.
We've used a some of Distributed storage (Longhorn, OpenEBS) things for Kubernetes and had no luck.
I agree with you at ALL the things you've listed. When we've started to used Ceph for the first time - (At that time I was an user, not a Manager) all the thing was a total mess. All the things are set to default and Use all JBODs on Node. (And I've started to face Corrupted WALs and OSDs came from nowhere.)
The handful of documentations was very helpful to cleans up and make it to run correctly.
Maybe, when I go back to the starting point, I'd like to say "move to cloud" or "do nothing" or following your guidance.
I should have been more gentle in my statement, for that I apologise.
if you do start again _and_ move to the cloud, AWS's lustre interface (FSx) is good enough (if expensive) or EFS, although that has slower metadata performance. Its good enough at RW iops though, assuming you monitor it correctly.
I feel your pain, and sympathise deeply. Best of luck!
It's quite simple in front, simple to explain that split object and metadata storage.
but it's too hard to select and tune each storage.
So I (or We)'ve decided to move to Ceph.