That's a single copy of all tweets. Not fanning out to up to 100 million inboxes. Completely different problems.
You keep citing raw hardware speeds of a single machine, yet we're talking about the feasibility of a distributed system being able to sustain random bursts of write amplification factor of 100 million across a decentralized database, with ideally exactly-once write semantics even if a failure occurs mid-way -- and that's all in addition to whatever the normal baseline write activity of all "normal" users with more reasonable follower counts. Again, completely different problems.
> I don't think so. Not with tweets.
So in your design, if the singular data center that maintains all users' inboxes goes offline for a long period of time, the entire product just goes down. And you think that's acceptable for a business valued in the tens of billions of dollars?
You seem absolutely convinced that a massive social network can be run on a shoestring budget with tiny staff, and no amount of evidence from someone like me (who actually worked on this stuff in depth, and posts with my real name, and expertise in profile) will convince you otherwise, so I suppose I should just stop replying to you.
> yet we're talking about the feasibility of a distributed system being able to sustain random bursts of write amplification factor of 100 million across a decentralized database, with ideally exactly-once write semantics even if a failure occurs mid-way
I think my assumptions are just a lot more relaxed than yours. This isn't a trading platform I don't see why you need exactly-once write semantics.
There are only 6 users with 100m+ followers and they avg a lot less than a daily tweet. @BBCWorld is #50 and it drops to 38m accounts. #1000 has 2 million followers.
> And you think that's acceptable for a business valued in the tens of billions of dollars?
Elon by his own admission grossly overpaid for it. Twitter has hardly ever eeked out a profit, it is not worth tens of billions of dollars. Nothing much would happen if it went down for a bit except maybe bored journalists would report on it thus, as ever, driving even more users to the website. But that is neither here nor there.
More to the point: if Elon and Obama and Bibster tweeted in the same minute (what are the odds) you would, gasp, have to stagger the fan out of the updates. That's alright too, for Twitter. It isn't really actually real time.
Those follower counts are also grossly inflated and as you understand yourself only a small fraction of them are online using the app at the same time as the person is tweeting. By the time they do check they might never even see the tweet.
To the people offline you don't need to fan out in a timely manner.
In short I believe the write amplification is much closer to 1 million than 100 million even with the pathological cases. And beefy enough hardware can handle those peaks.
Here's another way to think about it: Elon has 118m followers and just posted twitter has 260m daily average users. He is a bit like Tom from MySpace, half the users on the website are subscribed to his updates (not exactly really but for simplicity).
I think it is perfectly alright if it takes a full minute until all those users see his latest meme. It is very unlikely that even a quarter of all his followers are using the app during that exact minute, so we're talking 30m writes in 60 seconds. Big whoop.
> You seem absolutely convinced that a massive social network can be run on a shoestring budget with tiny staff
I would bet a budget of say <$1 billion/year and 100 engineers for the core functionality as is.
> no amount of evidence from someone like me (who actually worked on this stuff in depth, and posts with my real name, and expertise in profile) will convince you otherwise
Neither one of us presented any evidence, just opinions as outsiders, as part of an informal conversation. An appeal to authority isn't an impressive argument, I am also from this industry and with similar experience.
There is no need to take things personally. I think we just have a very different estimation of just how much activity twitter sees at peak and how strict the requirements are.
> I don't see why you need exactly-once write semantics.
World leaders use Twitter. It's a major international one-to-many communication platform. If tweets are lost or duplicated, it makes the platform look unreliable (because it literally would be) and as well as potentially making the tweeter look incompetent for posting twice. World leaders don't like to look incompetent, that can cause really bad things to happen...
> @BBCWorld is #50 and it drops to 38m accounts. #1000 has 2 million followers.
Even a write amplification factor of 100,000 is extremely problematic for the fully-materialized inbox model. A lot of prominent twitter users have followings larger than that.
> To the people offline you don't need to fan out in a timely manner.
So now you're adding additional systems on top, in order to scale. That's good, I guess you're starting to see that the problem is more complex than just spraying out every tweet to every follower's inbox. Now consider that when you actually build and scale a system like this, you'll need to keep doing that in a bunch of different areas, and the complexity keeps snowballing.
> And beefy enough hardware can handle those peaks.
There's no way to fit every users' fully-materialized inbox feed on one machine, so we're definitely talking about a large distributed storage tier / database here. Will you use "beefy" hardware for every single shard of your inbox storage tier?
> It is very unlikely that even a quarter of all his followers are using the app during that exact minute, so we're talking 30m writes in 60 seconds. Big whoop.
Once again, this really isn't like doing 30m write ops on a single box. It's queueing the writes via RPCs across a huge storage tier, while also needing some way to handle timeouts, retries, failovers on either side of the operation. All while the "normal" background level of thousands of tweets per second is happening from everyone else.
> An appeal to authority isn't an impressive argument, I am also from this industry and with similar experience.
> There is no need to take things personally.
I've literally built a reverse-chronological social network activity feed implementation, which successfully scaled to over 110 million posts/day. (For sake of comparison, Twitter was around 500 million tweets/day at that time, so this was def smaller than Twitter, but still quite large.) It did not use an inbox model. Took many months of my life, some of the most rigorous work I've ever done. My teammates and I evaluated several alternative designs, including fully-materialized inbox, running all the numbers in depth and building several prototypes. The takeaway was that a naive fully-materialized inbox would be completely and ludicrously infeasible in terms of necessary hardware footprint.
Separately, I've also spent years working on database infrastructure at extreme scale, including one of the largest relational database footprints on earth. I have a very good sense of what this requires. Yes, I'm posting "opinions", but they are based on many years of direct personal expertise.
Scaling a social network involves a massive number of challenging problems. Faster hardware doesn't magically make these problems go away. And while I haven't worked at Twitter, up until this month I knew four infra/backend engineers working there, and they're some of the best engineers I've ever known in my 17 year career.
I'm taking your comments personally because your comments are offensive. You're blindly saying I need to "refresh [my] assumptions" about a topic I'm literally an expert in. You're claiming Twitter could use some completely asinine overly-simplistic feed model, as if no one else ever thought of that, which would strongly imply every infra engineer at Twitter must be an idiot. In another subthread on this page, you wrote "The job cuts are clearly justified because of the extremely toxic work culture / cult" and it is necessary to "replace every single person who worked there and the entire tech stack". Seriously, WTF? These are hard-working humans with lives and families, they don't deserve this shit from their employer, and certainly not from offensive pseudonymous randos who have no idea what they're talking about. Have some empathy.
That's a single copy of all tweets. Not fanning out to up to 100 million inboxes. Completely different problems.
You keep citing raw hardware speeds of a single machine, yet we're talking about the feasibility of a distributed system being able to sustain random bursts of write amplification factor of 100 million across a decentralized database, with ideally exactly-once write semantics even if a failure occurs mid-way -- and that's all in addition to whatever the normal baseline write activity of all "normal" users with more reasonable follower counts. Again, completely different problems.
> I don't think so. Not with tweets.
So in your design, if the singular data center that maintains all users' inboxes goes offline for a long period of time, the entire product just goes down. And you think that's acceptable for a business valued in the tens of billions of dollars?
You seem absolutely convinced that a massive social network can be run on a shoestring budget with tiny staff, and no amount of evidence from someone like me (who actually worked on this stuff in depth, and posts with my real name, and expertise in profile) will convince you otherwise, so I suppose I should just stop replying to you.