Mastodon is written in Ruby on Rails and there are some inherent performance issues with that, it generates a huge number of Sidekiq jobs that can bog down a server quite easily. There are other, non-Ruby implementations aiming for compatibility with the Mastodon API though, so I’m curious to see how it will all shake out.
People can make a case for the developer productivity benefits of rails to a startup or business, but it’s hard to see it as worth the cost to the Mastodon community as a whole. But maybe it’s won the Fediverse market because of the depth of features which is a benefit of that productivity.
My read is it's mostly social, there's a lot of people accustomed to Mastodon and not much interest in exploring other options. There are implementations in Elixir (Pleroma, Akkoma) and work being done in Rust (Calckey, currently node but moving towards a Rust implementation). Mastodon dev team are not particularly open to criticism so my general sense is that admins should choose another project.
Yes, I’ve explored all this but I think Mastodon has won mainly because of its admin functionality, a huge fraction of dev effort is focused on tools to manage content moderation, spam policies and so on. Pleroma has a bad reputation because it was behind on that stuff at one time - maybe still - and a lot of instances had to be blocked because they couldn’t be managed effectively enough. I think it’s easy to underestimate how wide that moat is.
That's fair, I've heard complaints that mastodon's moderation tools also leave something to be desired but it might still be the best. I haven't been an admin so I can't comment firsthand.
FWIW, there are some - maybe one or two - alternate implementations of Mastodon server brewing out there. Not of a generic ActivePub server, but specifically a server with an intention to be fully compatible with Mastodon.
So in the future, there may be more, hopefully more efficient choices.
While Rails doesn't help, it also isn't really the problem here. The problem is a mix of deployment instructions that are complex and doesn't emphasise the need for robust caching enough (this should be behind a properly configured Nginx cache, and the entire site also ought to be behind a CDN), combined with a Mastodon-specific architecture that as you say is really aggressively generating async jobs. Mastodon is really unnecessarily heavy to run.
