The other key thing you lose is your identity. In the fediverse, your identity is your Webfinger handle, ie @user@server.com. Your server is literally part of it. Sure, you can migrate to @user@newserver.com, and keep the username part, but your identity still changes.
Truly portable identity via DIDs, ie you can keep your underlying identity even if you migrate servers, is one of the key reasons the Bluesky team made their own protocol. https://atproto.com/guides/faq#why-not-use-activitypub
The webfinger handle does not need to be on the same domain/hostname as the Mastodon server. E.g. not on completely different domains, but for Mastodon it makes no difference, but my personal Mastodon install is on m.galaxybound.com but my webfinger handle is on galaxybound.com.
And there was no need to make a new protocol for a portable identity - a change to ActivityPub to support did's as actor urns would be sufficient, and would also open the door to unilateral account migration fairly easily.
This is my big problem with Bluesky - all of their gripes about ActivityPub would be easily solved in ways that'd make interop a temporary problem of getting people to buy into protocol tweaks, instead of inventing something from scratch.
Their claim that it's not easily possible to retrofit e.g. did's and signed repositories onto ActivityPub makes me question whether they understand ActivityPub at all, because there's nothing about it that would be problematic. E.g. objects are already signed - their mechanism for migration would require some changes to the signing mechanism to allow users to make a unilateral assertion that the key on their new instance belongs to them, but not much more. DID's is down to how ActivityPub clients and servers resolve URLs, nothing more.
You wouldn't even need everyone to buy into these changes - the worst case would be lack of federation w/instances that fail to support it - in other words no worse than starting your own network.
Even then it'd be possible to maintain fairly broad interop by announcing the did's in ways that'd allow also specifying resolvable urls to a proxy.
So, the thing that made Mastodon click for me was when I reminded myself social media doesn't matter. That's really the whole point of this stuff to me. Twitter's long failure began when people were convinced they should make it more important in their lives than it ever deserved to be. When the daily news started to earnestly read nonsense like "X tweeted in response …" for minutes on end, or when the RCMP used Twitter to communicate important public safety information, I felt it in my bones (though I might not have understood exactly why): this was wrong.
I set my Mastodon posts to auto-delete after 6 months so I don't care if I lose them, and I made sure to have some "me" links in the appropriate places pointing at my profile. Even if I were to lose that paper trail from doing proper account migrations, it's pretty easy to say which profile is mine. And if I lose followers, so be it. I'll follow people when I remember who they are. I'll make new posts. If I don't, that's fine. Life happens.
You can indirect webfingers, so @you@site-you-own points to @you@someone-else-owns-this-server.
It's not ideal because there isn't a reverse link; if someone-else-owns-this-server dies, people who were following you on it will see you evaporate. But you can edit @you@site-you-own to point to @you@your-new-site so that at least people holding your ur-name can find you.
But unfortunately, there are few better solutions when someone else owns the data. One nice thing about the Fediverse is you can ameliorate most of this by setting up your own server (though I won't pretend that's going to be the solution that replaces everyone using Twitter; I maintain my own server and it is the same pain in the ass that self-hosting has always been).
The better solution is for implementations like Mastodon and Lemmy to let users bring their own custom domains, and have robust data export APIs. Then even if your home instance disappears overnight you can migrate to a different instance.
Truly portable identity via DIDs, ie you can keep your underlying identity even if you migrate servers, is one of the key reasons the Bluesky team made their own protocol. https://atproto.com/guides/faq#why-not-use-activitypub