And who will pay for that openness and decentralization? Let’s hypothetically say that Twitter is closed, millions of users discover Mastodon and move. Mastodon instances will be down in matter of seconds. How do you approach this? By volunteers adding more instances(that they can close anytime)? This will not change anything. Everything cost money and living in an “free” world bubble isn’t helping in any project adoption.
So I do not see any advantage in federated system. It’s cool as technology and all, but completely unprepared for huge traffic or real life scenarios.
PS. Please do not say anything about “anyone can start his own instance”. No, average Twitter/Facebook consumer can’t start his own instance.
Did Web 2.0 make us all forget how open IRC networks were run?
Resources donated by an organization in the form of a server linked into a larger network, a committee that vetted new server applications to the network, volunteer administrators for the network and the individual servers, coordinated regional and global upgrades. And as network users increased, reforming under a hub and spoke models to improve scale and capacity.
And when a single IRC server went away after some time operating for its various reasons, the network kept going.
Could the average IRC user start/host their own instance *and* link it to the larger network? No. But they didn't need to.
As a daily IRC user I can say this: IRC never was reliable. Constant attacks, splits, nick squatting or other crap ware pain in the ass. That’s why it never was adopted as a mainstream communication platform, and at its peak it had maybe 400k+ users. Now most of that are bots and stale sessions.
I am a daily IRC user and I think you are exaggerating the problems. I can count on my one hand the number of times I have seen an attack on my nick or the channels I hang out in, in the last 15 years. Those attacks pass without much disruption (sometimes requiring staffer intervention). Nick squatting is solved by Nickserv these days. Splits do happen occasionally but they resolve on their own automatically without much disruption.
It is ok if it never gets adopted as mainstream communication. But for the target audience (like opensource support communities being the target audience of Libera), it works quite well.
in the last 20 years I have had more attacks on my email account and $whatever_current_social_network account then I had on my irc account.
Things like nickserv and ip cloaks (which have been part of nearly all networks I used to connect to and those I connect to) do the job just fine.
Is it easy to flood down a server, sure is, but if there's more then 1 hub on the network, things will settle pretty quickly normally.
In my opinion, the main reason why it was never adopted as a mainstream platform, is because it was never picked up by a big corp that saw a way to earn money off of it.
I don't think we can just assume that "reliable" is a globally agreed property to judge alternatives. That property is something that came about and on some level spoiled users. Because for commercial social network providers any downtime meant a loss of users, eyeballs, and most importantly money.
Insert any modern chat app and random user can drop a screenshot there trivally, do some basic text formatting, and paste code in nicely colored blob of text, all from comfort of single login in a browser. And now even jump to voice chat or video conference directly from it.
They can also look at what they missed in channel by just scrolling up. Sending files just worked. You just got notifications without extra fuckery. Hell, you get digest with conversations you miss.
That's why modern closed down chat ecosystems won. Coz it's easy for "normies" to use.
Let's take other example, XMPP. It had all that but in network of XEPs that some clients/servers implemented, some not, some just not very well, and some being fucky to config. It wasn't just "login here and get every feature" like it is in modern chat clients.
It had it's one shining moment where both facebook and google offered XMPP way into their garden so you could have connectivity across federation and just have a single contact list but that didn't matter to non-nerds that just used facebook/google talk so once they decided to close their gardens any benefit of xmpp faded away.
Survival is not the issue. Mastodon will survive. Tumblr survives. Even MySpace survives. But major disruptions tend to lose users.
(And yes it's true that the potential Twitter acquisition is a potential major disruption. But it's not going to be shut down after a $44 billion investment.)
The staffers setup the new servers and did all the heavy-lifting.
As a user, I only had to point my client from Freenode to Libera (exactly one line change in my client config), run /msg nickserv register to register myself, run /msg chanserv register to register the channels I op-ed, and it was all done.
Total time spent was less than 30 minutes. The next few days, others did the same and the community started trickling in to the channels in the new servers. Seems seamless enough to me. I doubt such an easy migration is possible if Twitter disappears suddenly.
And yet userbase got decimated for most channels when moving from Freenode to Libera. Just because it was only 30 minutes (for you or for anyone) doesn't mean people will go through the effort.
The active userbase was not affected very much. (Note that some channels moved to OFTC, not Libera.) The lurker userbase was more than decimated – I think it about halved – but they were barely part of the communities, and there might not even have been anyone sitting behind the IRC clients.
anecodote only, but #ardour lost precisely zero users when moving from freenode to libera. just because people on channels you joined weren't willing to go through "the effort" doesn't mean that other people feel that way.
It's not $44B that twitter will have available and can spend. Most (all?) will go to current investors to buy their stocks at a set price, which is where the $44B comes from
It's probably more about how much the new owners will want to drop into it and how long before it moves to x.com (?) and becomes an everything app
> It's not $44B that twitter will have available and can spend. Most (all?) will go to current investors to buy their stocks at a set price, which is where the $44B comes from
Why did you feel the need to mention this 100% obvious fact?
Of course I meant that the new owners wouldn't shut down something they just spent $44 billion on, thereby throwing their investment in the trash, not that Twitter would magically get a $44 billion operating cash infusion.
> By volunteers adding more instances(that they can close anytime)?
"That they can close anytime", just like twitter in this example. Partly, yes. But that sort of total exodus would mean a lot of additional people contributing ideas and code to the Fediverse, not just servers, but by making it easier to run your own instance. Who's to say it couldn't be run just like an email client with the right ideas and effort? It's such an extreme example that I'm not even sure it's useful to discuss.
What traffic is it prepared for? It would be interesting for you to provide the numbers and the evidence which backs this up.
As for real life scenarios...there are upwards of a million people using it right now. I've made friends, networked professionally and found several homes there. I am literally a real life scenario and so are the people behind most of the posts there.
And today you are right about "anyone can start their own instance", but it's a darn sight easier than running your own twitter.com, and it'll get easier every year.
> "That they can close anytime", just like twitter in this example. Partly, yes.
Twitter pays people money to keep their service running, so there's that incentive.
> As for real life scenarios...there are upwards of a million people using it right now. I've made friends, networked professionally and found several homes there. I am literally a real life scenario and so are the people behind most of the posts there.
Twitter is, say, 300m MAUs. That would mean the volunteer Mastodon infra would have to increase 300x (assuming scaling is linear, and the Mastodon community hits Mastodon as hard as Twitter users hit Twitter) to cope with similar traffic numbers.
> But that sort of total exodus would mean a lot of additional people contributing ideas and code to the Fediverse, not just servers, but by making it easier to run your own instance.
Most users aren’t interested in contributing anything to the platform. Social media platforms popularity lays in simplicity. No one wants to run anything, just use service without any hassle.
> there are upwards of a million people using it right now.
Compare that to 200m+ users of Twitter sending 500m+ messages daily. I bet Mastdon can handle this without a sweat.
This right here. He's right you know. Everyone knows that Mastodon can't handle this amount of users and after 6 years, it's enough to see that by itself it has failed, (unless you count Truth Social as a great example of a Mastodon usage that has more users than Mastodon itself in less than a year)
It is not early days anymore and no non-technical user is interested in hosting their own servers for chatting with another person. They don't care about decentralization as even if they tried they will recentralize to the main Mastodon instance.
And hell, even tech users aren’t inclined to sign up for a sysadmin role for free with absolutely nothing in return except users berating you whenever there’s problems. Which there will be at some point in time.
Source: I ran chat services for friends. I no longer run chat services for friends.
> PS. Please do not say anything about “anyone can start his own instance”. No, average Twitter/Facebook consumer can’t start his own instance.
This isn't a law of the universe, it's just software people haven't written yet. Installing new client-side apps was hard, until it wasn't. "Anyone can start their own instance" will be easy once someone writes the software to make it so. (Presumably a cloud provider like AWS, since that's who stands to profit from lots of people wanting to run server-side apps)
Anyone can just pay $6/month to the good folks at masto.host to manage one for them. It's enough for them and possibly a few friends. I don't know if there are any other managed Mastodon companies, but this one has been around for years and has a good reputation. Their managed instances also meet the joinmastodon.org listing requirements by default.
Who said anything about the Fediverse having to be free ?
There is absolutely no doubt that should Twitter die, if no single actor can emerge quickly enough, for-profit actors will emerge and they will have all good reasons to be compatible with something that already exists. There will be mega large instances paid by siphoning data and with ads, there will be large instances paid by users/funds/donations, there will be small, community instances. Maybe HN will have its own instance; how much do you pay for HN today ?
As a counter example, email is a federated system too. I don't think a federated network should, or can for that matter, mimic the user friendliness of a closed system; so there won't be massive exodus of users from Twitter to the federverse, no matter how screwed Twitter became.
Our pay-what-you-can cooperative Mastodon instance at social.coop, running strong for over 5 years, is currently debating what to do with our 10,000€ budget surplus.
The idea that social media costs more to operate than people would be willing to pay is false. It's propaganda from the people who profit from keeping you trapped in their closed networks to monetize your attention.
Well, it was true decade, maybe two ago. Stuff got faster and therefore cheaper too.
Server written in reasonably fast software on $10 (well, probably some extra bux in storage if community is image-heavy) VPS can reasonably serve tens of thousands of users in typical forum/blog/hn-esque format. Mastodon isn't but I don't doubt someone would rewrite it in something faster if there was a need for it.
> Mastodon instances will be down in matter of seconds.
Of course not. Mastodon instances can be capped in the first place, and anybody with a rudimentary server management knowledge can start their own instances on a cheap server. Mastodon has current hundreds of instances, let's not pretend it can't go to thousands if the user base increases.
I think the problem is exactly that the average consumer can't start their own instance. What if there was a front-end service that made creating a fediverse instance as easy as creating a discord or Slack, and handled all the messy technical stuff with setting up an instance for the average user, while at the same time allowing said user to have full control of the cloud files? The front end would be incredibly light weight (just API calls, no data storage), so even if it shut down, as long as it is open source someone else could run their own instance of it on a different URL and the user could keep admin of their instance through that.
Admins of existing instances can configure user limits, close registration, etc., so new user will move to other instances or create demand for commercial instances.
> Mastodon instances will be down in matter of seconds. How do you approach this?
The same people who pay for everything right now will pay for it: us. Some instances will have patreon, others will be voluntary donation, others will use some craptocurrency, others will have contractual subscriptions, some will have ads... And whatever models are best will win out. Quit with the FUD. Just sit, back, relax and watch it happen.
Simple: It collapses and the millions move on to the next one, leaving the collapsed server to catch up and come back online. Like a scared but surviving turtle. Most servers are crowdfunded. Even the project's instances get funding through Patreon.
The official instance finding site seems to be good about spreading the load out every time Twitter burps. You have to meet certain reliability requirements to even be listed.
Is there anything to prevent a person/group from setting up a Mastadon instance with a charge to cover hosting, admin, & support costs (something like businesses charging for service on Open Source software support)? This could both make it more stable and sustainable and be a barrier to bots/trolls.
Nope - Mastodon supports invite-only which can aid in this sort of set-up; I'm sure other platforms do as well. And if one runs a close-knit community (which takes more than expertise and infrastructure), donations or something like a Patreon scheme can work.
This is why I view the "federated" form of decentralization to be more of an intermediate stop-gap between fully centralized and fully decentralized in the form of true P2P.
For a decentralized social network to be viable/sustainable (especially on the scale of something like Twitter), it has to be truly P2P, not federated on volunteer-run servers paid for through donations. That volunteer-run federated model is really only sustainable for smaller niche communities, not a global social network.
As of right now, the closest framework I can think of to handle something like this is a social network built on OrbitDB: https://github.com/orbitdb
Fully p2p doesn't really work well either. For a fully p2p network, nodes will have to hold the entire data.
Now first storage space taken will be ridiculous, and then you will be hosting content you don't want to host because it could be straight up illegal in your country.
Look at any crypto-blockchain and count how many full-nodes vs thin-nodes are running in networks.
Then there is the whole tragedy of commons: average seed ratios and streaming video players for torrent content shows that are not willing to contribute to the greater good.
Yeah, and there will be people that will just throw their porn collection there, hogging space on other people's drives for no good reason.
There is an option of dividing it between "publisher" (source) and "cachers" (caching whatever is recent and popular) but that only helps a little bit.
> Then there is the whole tragedy of commons: average seed ratios and streaming video players for torrent content shows that are not willing to contribute to the greater good.
Over 2 decades of pirating that has only been a problem for obscure and old stuff.
I don't see it being a problem in social network sense; you publish your stuff, if there is nobody wanting to look at it, seeds 1, leeches 0. Moment there is interest it propagates. It is of course problem if you want to see something archival that author is no longer in the network tho.
> Over 2 decades of pirating that has only been a problem for obscure and old stuff.
It's a common problem for old stuff like you said, very new stuff with specific formats. It's an issue for new stuff because it often doesn't have enough starting seeds, and people leaving as soon as they're done watching/downloading doesn't help the situation. Before I've discovered usenet this was a very common issue for me.
Then there is specific file formats like chunky movie rips with Dolby Vision + Dolby Atmos + Dolby Yet Another Audio Format + all of that again in 5 languages. People will often choose a different release to download, and that leads to fewer seeds.
I know a group of people on a popular torrent tracker that just downloading everything and seed forever. They do entirely for free. Those petabytes will probably rot once they...die.
edit: Scratch that, a more fleshed out and popular protocol that implements this P2P concept is Scuttlebutt: www.http://scuttlebutt.nz/
The most user friendly and refined client I’ve found that implements this protocol is Planetary: https://www.planetary.social/
Funny, I've discovered scuttlebutt, that you're talking about, while I was reading about cluster membership and fault detection protocol. I was so confused of how all of this chat rooms have to do with what I was reading about just a minute ago.
Your comment piqued my interest, but I didn't find the android (Manyverse) or desktop (Patchwork) apps to be user friendly. The UIs had a few UX oddities and unless you are convinced to join by others who are already using it, the social network is barren. Even trying to chain my from the most active people (last posts months ago) through their follower and following lists didn't fix my impression of it being a historical ledger.
I am not participant but I have seen some invite only fediverse instances. Can't there be paid instances too, even pay by (please don't hate me here) watching ads ? Does actually anyone need to cater to millions of users?
Absolutely. In Mastodon it's a pretty simply setting IIRC and I expect it's commonplace across other microblogging platforms that use ActivityPub. I certainly wouldn't be against the principle of joining an instance that was paid for.
The whole "millions of users" fallacy is the result of people not being able to grasp what federation is about. The network can easily accommodate millions of users. Individual instances don't need to be able to.
i.e. my laptop could handle the write load. A retired nerd with a real server they put in a data center for bandwidth could easily run that level of traffic on 2022 hardware.
For reference my work laptop (8 logical cores, so 4+HT? 32GB RAM) can handle 100k rows/second sustained inserts into postgres 14 with some batch jobs I'm working on. You can buffer http requests into batches and easily handle way more than 10k/s on a server while still providing synchronous semantics and reasonable latency to the client (e.g. flush batches every 10-100 ms).
I doubt Mastodon is designed for that kind of scalability, but most techies could probably afford to run Twitter as a hobby if they knew what they're doing and they weren't trying to do all the analytics and advertising stuff to monetize it/just wanted to provide the service.
I think you don't understand that 10,000 tweets isn't 10,000 inserts. It's not enough to just write tweet, you need to fan-out writes to every user's feed. It's also not batched into neat transactions like your batch jobs.
You don't need to fan out writes to feeds. Users subscribe to other users, not tweets. You can attempt to send out the notification to subscribed users, and if it fails, that's fine. You don't need to record notification status. Have a worker that records (in the db) the last tweet id it's processed, and just regularly joins N new tweets to authors to subscribers and attempts to send. You can play with how that query works to limit total work done in a chunk rather than N new tweets (in case of someone with many subscribers), but the idea is straightforward.
You can easily batch writes into neat transactions: make a queue, have your POST handler write (row, callback) onto the queue and await the callback. Have the queue reader grab a chunk, push it to the db in batch, and execute all of the callbacks on commit. The callbacks return a 200 to the client, or 500 if the commit failed. This can all happen fast enough to be done in "real time" (however fast you want your queue worker to flush batches).
You can do all of this in a couple dozen lines of code with something like Scala/ZIO.
Computationally, it is totally doable. The biggest constraint is the cost of storage.
I've worked in a social network with feeds. Feeds are hard. Both fan-out on write vs fan-in on read both have upsides and downsides. Either way, it's "cheap" problem to solve in a couple of dozen lines of code.
When I said that you can't batch, I meant that each tweet will be at least one transaction. Async write with batching like you suggested will have a horrible user-experience, also your client often is a mobile device or browser - how do you deliver a callback from server there?
Nah, how many inserts your laptop can make and how many tweets created in a unit of time are two irrelevant metrics.
> how do you deliver a callback from server there?
You don't. The callback is on the server before it ever responds to the request. The client sees a synchronous response with a delay of a few extra milliseconds.
Here, I slapped this together to demonstrate the technique[0]
Which processes up to 8k requests at a time, waiting up to 10 ms for a batch.
The worker does a bulk db insert, and completes the promises with the generated ids.
Similar techniques should work on read batching, but I haven't tried that. You can also speed that up some more with the COPY protocol, but IIRC you need to be more careful about escaping/SQL injection. The example I wrote uses prepared statements/parameter binding.
On my 6 year old mid-range desktop (this CPU[1] and this disk[2]) this program can process ~30k `create`s per second. For about $1500, I could buy a new computer with a Ryzen 9 7950 with 4x the core count/8x the thread count and 2x the single-threaded performance, so around ~10x more processing power, 128 GB of RAM, and a Samsung 980 Pro SSD, which can do 1M Write IOPS (25x more than my SSD) or 5GB/s sequential writes (10x more). So a $1500 computer with a single disk should be able to do around 300k/s. PCIe gen 5 is now coming out, which will allow for another doubling of disk performance.
128GB of RAM means you can keep at least 100M rows worth of index in memory. It's not that expensive (under $10k) to build a server with 1TB of RAM.
Totally feasible for a hobbyist to do without tons of tricky optimization (the code I posted is purely functional Scala!); people spend $20k on a jetski or $80k on a truck. Like I said, the most expensive part is going to be the storage, but you could do something like only store the most recent 1000 tweets per person, and charge $10 to bump that up to the most recent 10 million tweets or something. You'd come out at a substantial profit with that model if you got a few thousand takers. Similarly you could charge to let someone follow more than a few thousand people so you could pay for a read replica or two.
I'd assume it's not so much the peaks that are a challenge - most of these 10k tweets/second aren't critical to serve to anyone fast, and that scales horizontally- it's the hot spots, that one tweet thread in the spotlight right now that everybody wants to read and jump on - that doesn't scale by just adding more servers
Yes, hosting is a cost and it is not 'free', hence the frequent downtime with Mastodon instances, even when they had traffic during Elon's takeover of Twitter many of then could not even handle the new users.
Also, these users don't even know which instance to go to, since there is little to no-one to talk on there. If there are 'hundreds of thousands' of users then that means they have just recentralized on Mastodon.social, the "main" instance, defeating the point of it all.
> PS. Please do not say anything about “anyone can start his own instance”. No, average Twitter/Facebook consumer can’t start his own instance.
This is why Mastodon has failed in the first place after almost 6 years with this system.
Six years of steady growth is a Silicon Valley failure. It is not a failure by any reasonable measure. I predicted back in the early days that Mastodon would grow slowly and organically as more people figured it out and helped people in their circles come over. It's slow and steady, but I was right. This is how things grew before anything less than a double-digit billion sale to one of the big tech companies was seen as a failure.
I look at all of the interests and discussions and shitposts that are on my home and local feeds, watch the interactions between mutuals and people argue, learn, laugh and join in shared hobbies and simply cannot fathom how this can be "failed". All this and if I turned off my anti-virus I wouldn't see any adverts and am never subjected to abusive interests or dark patterns. How is this failure? It's what I want from the Internet. It's real, actual people.
Weren’t you complaining about mastodon having “no users” in the last big discussion about it here? Do you have some kind of personal issue with the protocol?
There are plenty of people using it. It has not “failed”.
> Weren’t you complaining about mastodon having “no users” in the last big discussion about it here?
Yes, I said: 'Little to no users'. After looking at it for a couple of years, it is not the typical twitter user that is self hosting their own Mastodon instance and just the same tech-folks that are doing that (unreliably) and sitting on Mastodon. The level of social interaction on Mastodon is so low and limited, that they still use their Twitter accounts more than their Mastodon accounts.
So yes, it is not early days anymore and we have given it enough time and it has already failed.
I'm following so many accounts that if I don't sign in for three or four days I can barely keep up with my home feed. Almost all of it is interesting, funny, insightful or simply chill discussion. They almost exclusively use the Fediverse, none of them use the Fedi as a second-class citizen. Your insistence that something that is alive, growing and healthy has "failed" is simply proof that you have failed to bother with it because of your preconceptions.
Exactly. I use mastodon daily, I follow tons of people that do as well. Mastodon doesnt need to be massive or fulfill whatever growth expectations armchair tech entrepeneurs expect of social media platforms here. It just has to be reasonably easy to maintain and actually play an important role in people's lives and it is absolutely doing that.
Also, I put content warnings when I blab on about some tech thing because not everyone is a techy there. I am friends with lots of people there who will roll their eyes and walk away if you start blowing their timeline up with that kind of topic and you arent conscientious. It isn't just techies all hanging out with no reason to be there other then the tech novelty of it, it is a lot of peoples' home.
> Your insistence that something that is alive, growing and healthy has "failed" is simply proof that you have failed to bother with it because of your preconceptions.
Having 90% of registered accounts inactive with only 10% of them actively using the platform isn't exactly 'alive', 'growing' and 'healthy' especially when they occasionally run back to Twitter since they know little social engagement goes on Mastodon. 10 is closer to 0, than 90 and usage is still declining; Hence "Little to no one".
But we both know it is not just that. Not only they can't help using Twitter more, they won't move to Mastodon for the exact same reasons as I said and Twitter's network effect, hence why little to no-one is using Mastodon. The same tech-folks like (Mastodon.technology) are the ones 'self-hosting' these instances and not the regular users, since they don't care enough to even use it.
Not even the one operating Mastodon.technology could handle it. Might as well recentralize back to Mastodon.social just to save itself from the very low levels of social interaction since Mastodon has already repeated the same problems as GNU Social once again.
> Having 90% of registered accounts inactive with only 10% of them actively using the platform isn't exactly 'alive', 'growing' and 'healthy'
That's not at all true. Account activity follows the Pareto principle. It's not at all unusual for any online service to have a large number of inactive users. Perhaps it's different for Twitter, but considering I've probably signed up for it three times and use it approximately never, I'm skeptical.
So I do not see any advantage in federated system. It’s cool as technology and all, but completely unprepared for huge traffic or real life scenarios.
PS. Please do not say anything about “anyone can start his own instance”. No, average Twitter/Facebook consumer can’t start his own instance.