About 2 days ago, I noticed that Facebook added multi-level reply threads for a small percent of posters, but as of this morning it seemed totally broken for me on the latest ios build. Seems unlikely to be related, but I wonder if it's possible they broke the site trying to hotfix it.
Regardless of the root cause, I'm feeling for the oncall folks whose Thanksgivings are getting ruined right about now.
I work pretty heavily with the Facebook Graph API and things have been pretty messed up ever since Cambridge Analytica and the scramble afterwards.
I've got a ticket from August about a basic piece of functionality - deleting a post via its ID - that's completely broken, and all I get is an every-few-weeks "should be fixed", "nope, it's clearly not", "ok stand by" exchange with them.
Basic stuff like "show me the user's Facebook pages" breaks for everyone for hours at a time. When we attended F8 they talked heavily about their intensive automated testing, but I don't see that much evidence of it. I suspect a lot of things just got turned off in the mad "shut down EVERYTHING" scramble and they're still working out the impacts.
Also curious if FB outages affect global internet traffic? Are there any relative stats? (E.g. if google to go down, it would have an economic cost comparable to a natural disaster)
There's several anecdotes internally at Facebook that get told to new starters about some instances of when facebook.com was down. One involves masses of people calling the police to report that facebook is offline (wat, why?), another involves so many people flocking to twitter to complain that fb is down that that crowd then knocked twitter offline ("How do you know if facebook is down? Check whether twitter is online" sort of thing).
I don't feel comfortable sharing the exact figures, but during my time there the cost of a few minutes of downtime on an ads-adjacent message queue cluster directly turned into millions in lost revenue, so it's safe to say that losing the capability to _serve_ ads (as opposed to charging for serving them) would have a pretty damn high economic cost to those advertisers. As Workplace gains traction, downtime there will creep into the "cataclysmic" side of the economic costs scale.
Did you count the lost revenue as revenue per minute times the number of minutes the site was unavailable?
If so, that's actually incorrect when calculating revenue loss. There will be actually be an uptick in traffic after the outage, either through people postponing their intended visit or checking if the site is down for them after it comes back.
I know that at LinkedIn we broke part of the advertisement campaign creation flow from downtime in our service for a short time, and the impact was negligible since ads were still being served, and new ad creation would be just be retried later by advertisers.
Now if you just lost the capacity to serve ads, there might be an impact, but since a lot of advertisements are display ads (to raise brand awareness over an immediate call to action), it's not clear whether or not it's a simple minutes down times impact per minute calculation.
Revenue/minute * duration actually worked here because what went down was the moral equivalent of a Kafka cluster — ads were still being served, but we couldn’t log clicks/impressions over that period of time
the scale FB operates on is staggering! Human factor, while still present, is very low. Wonder what could have happened there. They don't usually release postmortems, right? (since the general service they provide to the masses is "free", meaning minimum obligating to reporting. Would be curious to read the postmortem, analysis of what happened.
Nope, nothing to do with FreeBSD or Erlang. Can't say what it is, but there are hundreds of other differences between which pieces of FB infrastructure WhatsApp uses vs. what other products use. This is in one of those hundreds. ;)
Given most of America, except those unlucky people like me that are at work for their 14th Thanksgiving in a row, are hanging out with family not starting at the internet either watching parades or football instead of tracking political happenings that the vast majority of Americans are clueless about because we have no realistic way as an individual to do anything about even stuff at a town/county level, I'm guessing it has nothing to do with that whatsoever.
Regardless of the root cause, I'm feeling for the oncall folks whose Thanksgivings are getting ruined right about now.