What do you think are some gotchas/limitations when using mysql-ndb-cluster ?

It's become very stable with the years. It's somehow never penetrated the silicon valley echo chamber, probably due to instability in its earlier years. Operationally, it's got most things you need - on-line add node, rolling upgrades, monitoring, etc.

We use NDB to store both the metadata in-memory, but also to store small files on NVMe disks. We had talks with lots of other DB vendors, but, frankly, none of them have high performance support for cross-partition transactions, which is needed. DBs like VoltDB, MemSQL, NuoDB all have promise, but serialize cross-shard operations.

Have you explored using FoundationDB? It doesn't have a SQL interface (anymore), which might mean it isn't suitable if SQL is required.

Not yet, it's only recently been open-sourced. We have a startup now commercializing Hops, called Logical Clocks (of course), so we're busy with that.

The community is very helpful over on the forums (https://forums.foundationdb.org), so if you do end up having time to check it out, come on over! I'm sure the team at Apple (I'm not affiliated with them) would love to talk about it.

They're against coprocessors (having functions in the db) so you'll always be somewhat limited. Doing with shared-memory-etc will be hard.

Also you can shard many tables as hash, so most hot-path transactions be inside a server, which you can't guarantee with range-sharding.