Open anarchic communities are not toxic only while they are young and fresh. Eventually they will grow and either establish a set of formal and legally binding ceremonies with unsatisfactory result (actually I don't know any really good decentralized example here - Wikipedia is the closest one) or become toxic and die (IRC).
the point of mastodon communities is that they are not anarchic. The fact that they're human scaled and moderated is what allows them to impose sets of rules that actually make spending a few hours there palatable. (The same advantage that say, the smaller hackernews community has over a 500k population subreddit)
That's the advantage they have over twitter, they're able to set community rules that fit to the respective communities. It's twitter that suffers from the anarchy.
When twitter users are setting up filters and mass blocks to remove all the nazis and trolls and conspiracy theorists from their feed, they're essentially just trying to painfully recreate what you get in a mastodon sized community by design.
Can you give an example of mastodon community with clearly defined and legally binding policy, that protects rights of regular users? I could not find any (doesn't mean that they don't or cannot exist, of course) and highly doubt that any admin there has motivation or resources to develop it and properly implement. Pure declarations like "we are good fellas and don't allow abuse" mean nothing - they are not enforceable and regular user cannot sue instance admin for violations.
As a Mastodon instance admin, you just made an argument for me to never have a clearly-defined and legally-binding policy. Thanks.
The main thrust of my user agreement is "hi I am queer lady who is out of patience with people being jerks and wants a nice place to talk, if you piss me off you can go find a new instance or start your own". It's not much more formal than that, either.
Coffeehouse is heavily regulated. There's consumer rights protection, all sorts of safety inspections etc. The only thing that requires your trust in a coffeehouse is the quality of their coffee and the only damage you'll get with the breach of that trust is the price of a cup. Being banned from a social network harms you more than that.