Can't wait to find out what happened here. This seems to be a massive outage. Interesting how fragile things become when so much technology is concentrated into just a few companies.
My subjective impression as a web user since the late 90s is that now things break relatively rarely (I think it's the first time I have any such issue with DDG for instance) but when they do a huge chunk of the web becomes unreachable.
Back when things were more decentralized individual websites and services would have issues much more regularly because the individual software and hardware stacks weren't as robust and fault-tolerant, but then usually the problem would always be limited to a single website/service.
A Windows update restarted a critical server automatically. A core service is blocked from starting by a Candy Crush ad installed by the update. The Crandy Crush ad is somehow expecting a Copilot key to be pressed on the keyboard to let the system keep going.
MS engineers are waiting for an online purchase of a new $400 keyboard with the Copilot key to complete and planning to run to the data center to plug the keyboard.
However, the Bing outage is preventing the purchase to go through, because the payment somehow relies on Bing suggestions to load for obscure reasons.
Very very technically, if RDP is enabled and working, this could be fixed by rdesktop-ing to the machine from a Linux box and using xdotool to experiment with typing raw keyboard scancodes through the RDP session in the hope you figure out the encoding of the Copilot key.
I also appreciate that you prevented me from countering using a guard like "if RDP is enabled and working", and that a follow up answer actually provides the missing piece xD.
MS knows better than to run windows on their servers. They’ve famously been running Linux on their public web facing stuff for years, including when they were publicly discrediting Linux (because IIS and MS server were so good they couldn’t run their web services reliably and securely).
Update: the payment gateway is Stripe, which is not processing any transactions associated with the MS account. A developer has posted the issue to HN in the hope that a Stripe employee will see it and escalate the issue. /s
Bing does not work me at the moment (maybe they can service partial traffic? Unclear if it’s better/worse than a few hours ago when the outage started)
Genuine question, are distributed systems naturally more resilient?
I can see arguments for both sides. Your point and then the hidden failure modes without central observability and ownership. Nothing exists in isolation.
Not distributed per se, but diversity makes a huge difference in resilience.
When everybody is using the exact same tech, the fall out of an incident can be huge because it will affect everybody everywhere at the same time. Superficially it might seem efficient and smart, but the end result is fragility.
Diversity of species is what nature ended up with as the ultimate solution: the individual species do not matter, but life as a whole will be able to flourish. With technology, we're now moving the other way: every single thing gets concentrated into one of the few cloud providers. Resilience decreases, fragility increases.
I prefer heterogeneity rather than diversity. Different implementations of similar processes fenerally make different tradeoffs, incurring different bottlenecks, and resulting in an ecosystem with a higher statistical probability that one relative Black Swan won't wipe out a key structural function in it's totality.
It's actually a hallmark of building fault tolerant systems and ecosystems. Pity the economists and MBA's can't be convinced of it. Otherwise there'd be less push to create TBTF institutions.
Distribution alone doesn't make a system resilient. A distributed system can help with resilience for anything related to network or hardware failure, but even then you need to make sure the different resources don't have a hard dependency on each other.
If you want a resilient system redundancy and automatic failover systems are really important, along with solid error handling.
Think about a distributed data store for example. You may spread all your data across multiple distributed areas, but if each area is managing a shard of data and they aren't replications then you still lose functionality when any one region goes down. If you instead have a copy complete copy of data in each region, and a system to automatically switch regions if the primary goes down, your system is much more resilient to outages (though also more complex and expensive).
My point was just that resilience still depends on how a system is distributed and what else is done.
Distribution alone doesn't really make a difference, though pairing it was redundancy and failovers is going to get pretty far.
The case of mastodon.social is really a question of whether the value there is the network and protocol itself or the user created content posted there. If its the user content, the value is lost when the one host goes away. If the value is the network and protocol then yes, the value of the network is still there even though the data is gone. It does raise an interesting question of whether Mastodon is really considered distributed or not, the network is and hosts are using a shared protocol but the data isn't really distributed.
Yes there is the question of network vs data :).
And as you mention while some data end-up being distributed with Activity Pub the protocol is not made to allow restoration.
One point I find interesting too is that distributed network often allows more agency to external actors. For example if you believe that the resiliency of the mastodon.social instance is not enough for you then you can decide to host you own server with your preferred criteria.
That's really where ActivityPub starts to rub me the wrong way. Server admins really need moderation power since everything is hosted on their hardware, but it also is a poison pill for decentralization.
I can host my own server and make my own rules, but every other admin can just ban my instance.
I feel like that's actually a counterexample. At least most people with mastodon.social as their home server will probably not have a backup of their followed/following graph and never be able to recover.
With a large number of small providers, more often than not some of them will fail on any given day, but stars need to align really well to get a half-of-the-internet-is-down kind of failure caused by AWS or Cloudflare.
Not exactly “more resilient”, but rather, “the only way to gain more resiliency over a single system”.
A distributed system can be more resilient, but it also adds complexity, making it (sometimes) less reliable.
A single system with a lot of internal redundancy can be more reliable than a poorly implemented distributed system, which is why at a smaller scale it’s often better to scale vertically until a single node can’t handle your needs.
Distributed systems are more of a necessity than “the best way”. If we could just build a single node that scaled infinitely, that would be more reliable than a distributed system.
“A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.” — Leslie Lamport, 1987
Distributed systems with tight coupling and no redundancy are less resilient. It's not so much a question about distribution but more about redundancy and coupling.
RAID1 is mirrored. That is not what I would call a typical distributed system. It is a very redundant system. Like a cluster.
A distributed system without redundancy would rather be something like data stripped across disks without parity.
And that actually makes it less resilient, because failure of one component can bring down the whole system and the likelihood of failure is statistically higher because of the higher number of components.
When I think of distributed systems, the RAID1 analogy seems much more applicable than RAID0.
The term "distributed" has been traditionally applied to the original design of the TCP/IP protocol, various application-layer protocols like NNTP, IRC, etc., with the common factor being that each node operates as a standalone unit, but with nodes maintaining connectivity to each other so the whole system approaches a synchronized state -- if one node fails. the others continue to operate, but the overall system might become partitioned, with each segment diverging in its state.
The "RAID0" approach might apply to something like a Kubernetes cluster, where each node of the system is an autonomous unit, but each node performs a slightly different function, so that if any one node fails, the functionality of the overall system is blocked.
That second approach seems more consistent with what we traditionally label as "distributed" -- for example, the original design of the TCP/IP protocol, along with lots of application-layer protocols like NNTP and IRC, have each node operating autonomously but synchronized to other nodes so the whole system approaches a common data state. If one node fails, the other nodes all continue to operate, but might cause the overall system to become partitioned, leading to divergent states in each disconnected segment.
The CAP theorem comes to mind: the first approach maintains availability but risks consistency, the second approach maintains consistency but risks availability. But the second approach seems like a variant implementation strategy for what is still effectively a centralized system -- the overall solution still exists only as a single instance -- so I usually think of the first approach when something is described as "distributed".
You're assuming a stateful system where the state is distributed throughout the components of the system. For a stateless component of a distributed system, you don't need redundancy to recover from an outage.
>likelihood of failure is statistically higher because of the higher number of components
Yes, absolutely true, but resiliency for a distributed system is not necessarily like your example of data stripped without parity, unless we're specifically talking about distributed storage.
The controller isn't stateful; it's just an interface to the disks. If the controller fails, but the disks haven't, then all you've lost is the time it takes to plug the disks into a new controller.
With RAID1, there's also nothing specific to the RAID configuration inherent in the way the data is encoded on the disk. You might have to carefully replicate your configuration to access the filesystem from a failed RAID0 array, but you can just pull and individual disk out of a RAID1 array and use it normally as a standalone disk.
You will have a better chance at uptime with a RAID than a single drive so you hopefully don't have to climb up ventilation ducts, walk across broken glass, and kill anyone sent to stop you on your quest to reconnect those cables that were cut.
