> Once Microsoft's eBPF support for Windows becomes production-ready, Windows security software can be ported to eBPF as well.
This doesn’t seem grounded in reality. If you follow the link to the “hooks” that Windows eBPF makes available [1], it’s just for incoming packets and socket operations. IOW, MS is expecting you to use the Berkeley Packet Filter for packet filtering. Not for filtering I/O, or object creation/use, or any of the other million places a driver like Crowdstrike’s hooks into the NT kernel.
In addition, they need to be in the kernel in order to monitor all the other 3rd party garbage running in kernel-space. ELAM (early-launch anti-malware) loads anti-malware drivers first so they can monitor everything that other drivers do. I highly doubt this is available to eBPF.
If Microsoft intends eBPF to be used to replace kernel-space anti-malware drivers, they have a long, long way to go.
Yes, we know eBPF must attach to equivalent events to Linux, but given there are already many event sources and consumers in Windows, the work is to make eBPF another consumer -- not to invent instrumentation frameworks from scratch.
Just to use an analogy: Imagine people do their banking on JavaScript websites with Google Chrome, but if they use Microsoft Edge it says "JavaScript isn't supported, please download and run this .EXE". I'm not sure we'd be asking "if" Microsoft would support JavaScript (or eBPF), but "when."
This assumes eBPF becomes the standard. It's not clear Microsoft wants that. They could create something else which integrates with dot net and push for that instead.
Also this problem of too much software running in the kernel in an unbounded manner has long existed. Why should Microsoft suddenly invest in solving it on Windows?
Microsoft has invested in solving this for at least two decades, probably longer. They are just using a different (arguably worse) approach to this than the Unix world.
In Windows 9x anti-malware would just run arbitrary code in the kernel that hooked whatever it wanted. In Windows XP a lot of these things got proper interfaces (like the file system filter drivers to facilitate scanning files before they are accessed, later replaced by minifilters), and the 64 bit edition of XP introduced PatchGuard [1] to prevent drivers from modifying Microsoft's kernel code. Additionally Microsoft is requiring ever more static and dynamic analysis to allow drivers to be signed (and thus easily deployed).
This is a very leaky security barrier. Instead of a hardware-enforced barrier like the kernel-userspace barrier it's an effort to get software running at the same protection level to behave. PatchGuard is a cat-and-mouse game Microsoft is always loosing, and the analysis mostly helps against memory bugs but can't catch everything. But MS has invested a lot of work over the years in attempts to make this path work. So expecting future actions isn't unreasonable.
This is a weird reading of history. Microsoft has spent tons of effort getting as much code out of the kernel as possible: Windows drivers used to be almost all kernel-mode, now they're nearly all in userspace and you almost never need to write a kernel-mode Windows driver unless you're doing something with deep OS hooks (like CS was, although apparently even that wasn't actually necessary). The safeguards on kernel code are for the tiny sliver of use cases left that need it, it is not Microsoft patching individual holes on the leaky ship.
They haven't yet gone as far as Apple in banning third-party kernel-mode code entirely, but I wouldn't be surprised if it's coming.
A thing I think a lot of people don't include in their premises about Crowdstrike is that they're probably the most significant aftermarket endpoint security product in the world (they are what Norton and McAfee were in 2000), which means they're more than large enough for malware to target their code directly, which creates interesting constraints for where their code can run.
I'm not saying I'd run it (I would not), just that I can see why they have a lot of kernel-resident code.
Microsoft made the reasonable point that locking 3rd parties out of the kernel might have resulted in legal challenges in the EU [0]. It is an interesting case where everyone is certain in hindsight that they would have been ok with MS blocking access, but it is less obvious that they would have taken that view if MS had pressured a bunch of security products out of the kernel with no obvious prompting.
The 2009 agreement with the EU mentioned in the article seems to be the one about the integration of the Internet Explorer (IE) into MS Windows.[1] But it only applied to IE and the commitment was limited to 5 years.[2]
Or is the article referring to something else?
I see no reason why the EU should object to Microsoft's adoption of eBPF as long as MS Defender simply uses the same API that is available to all competitors.
Apple took the lead on this front. It has closed easy access to the kernel by apps, and made a list of APIs to try and replace the lost functionality. Anyone maintaining a kernel module on macOS is stuck in the past.
Of course, the target area of macOS is much smaller than Windows, but it is absolutely possible to kick all code, malware and parasitic security services alike, from accessing the kernel.
The safest kernel is the one that cannot be touched at runtime.
I don't think Microsoft has a choice with regards to kernel access. Hell, individuals currently use undocumented NT APIs. I can't imagine what happens to backwards compat if kernel access is closed.
Apple's closed ecosystem is entirely different. They'll change architectures on a whim and users will go with the flow (myself included).
But Apple doesn’t have the industrial and commercial uses that Linux and Windows have. Where you can’t suddenly switch out to a new architecture without massive amounts of validation costs.
At my previous job they used to use Macs to control scientific instrumentation that needed a data acquisition card. Eventually most of the newer product lines moved over to Windows but one that was used in a validated FDA regulated environment stayed on the Mac. Over time supporting that got harder and harder: they managed through the PowerPC to Intel transition but eventually the Macs with PCIe slots went away. I think they looked at putting the PCIe card in a Thunderbolt enclosure. But the bigger problem is guaranteeing supply of a specific computer for a reasonable amount of time. Very difficult to do these days with Macs.
Good luck getting that through a regulated company’s Quality Management System or their legal department. Way too much business risk and the last thing you want is a yellow or red flag to an inspector who can stop ship on your product until all the recall and remediation is done.
See Satya Nadella has recently said that Microsoft will now put security above any other value at Microsoft. He specifically even singled out backwards compatibility.
Microsoft is a big boat. It takes a long time to turn if the captain orders it. But the captain has ordered it. I expect the sacrosanct nature of back compat to eventually die. Windows will never turn into the moving target that macOS is, but I expect a lot of old stuff to just stop working, when security concerns dictate they should be removed.
> The safest kernel is the one that cannot be touched at runtime.
Can you expand what you mean here? Because depending on the application you are running, you will need at least talk with some APIs to get privileged access?
Yeah, Apple doesn’t allow any user code to run in kernel mode without significant hoops (the kernel is code signed) and tries to provide a user space API (e.g. DriverKit) as an alternative for the missing functionality.
...just like they did with Kerberos! And just like with Kerberos they'll define a standard then refuse to follow it. Instead, they will implement subtle changes to the Windows implementation that make solutions that use Windows eBPF incompatible with anything else, making it much more difficult to write software that works with all platforms eBPF (or even just its output).
Everything's gotta be different in Windows land. Otherwise, migrating off of Windows land would be too easy!
In case you were wondering what Microsoft refused to implement with its Kerberos implementation it's the DNS records. Instead of following the standard (they wrote!) they decided that all Windows clients will use AD's Global Catalog to figure out which KDC to talk to (e.g. which one is "local" or closest to the client). Since nothing but Windows uses the Global Catalog they effectively locked out other platforms from being able to integrate with Windows Kerberos implementation as effectively (it'll still work, just extremely inefficiently as the clients won't know which KDC is local so you either have to hard-code them into the krb5.conf on every single device/server/endpoint and hope for the best or DNS-and-pray you don't get a Domain Controller/KDC that's on an ISDN line in some other country).
This to me ascribes way too much to mustache-twirling villainy at Microsoft, but to me fails to account for the fact that engineers surely make many of these implementation-detail decisions. These engineers aren’t incentivized to create lock-in. I think it’s more likely that sometimes for a feature to play well with other existing parts of the Windows ecosystem, compromises are made to the standards-compliance. Microsoft may have shipped those related interfaces before this standard had been hashed out, so they have a choice to break everything or to not perfectly follow the standard.
Note: I’m not a Windows dev so I can’t speak to specifics of anything like your Kerberos example. I just don’t believe MS is full of evil engineers, nor that Satya Nadella visits cubicles to promote lock-in practices.
> These engineers aren’t incentivized to create lock-in.
Ever heard of something called “money”?
> I think it’s more likely that sometimes for a feature to play well with other existing parts of the Windows ecosystem, compromises are made to the standards-compliance.
So you're basically saying that you're too young to remember the “good” old days of Embrace, Extend, Extinguish, right...?
This doesn't really seem like their strategy anymore. It's not like Edge directly interprets Typescript, for example. While they embraced and extended Javascript, any extinguishing seems to be on the technical merits rather than corporate will.
In the case of security scanners that run in the kernel, we learned this weekend that a market need exists. The mainstream media blamed Crowdstrike's bugs on "Windows". Microsoft would likely like to wash its hands of future events of this class. Linux-like eBPF is a path forward for them that allows people to run the software they want (work-slowers like Crowdstrike) while isolating their reputation from this software.
> Why should Microsoft suddenly invest in solving it on Windows?
If they can continue to avoid commercial repercussions for failing to provide a stable and secure system, then society should begin to hold them to account and force them to.
I’m not necessarily advocating for eBPF here, either. If they want to get there through some “proprietary” means, so be it. Apple is doing much the same on their end by locking down kexts and providing APIs for user mode system extensions instead. If MS wants to do this with some kind of .net-based solution (or some other fever dream out of MSR) then cool. The only caveat would seem to be that they are under a number of “consent decree” type agreements that would require that their own extensions be implemented on a level playing field.
So what. Windows Defender shouldn’t be in the kernel any more than CrowdStrike. Add an API. If that means being able to send eBPF type “programs” into kernel space, cool. If that means some user mode APIs, cool.
Not necessarily disagreeing with you, but as far as 'avoiding commercial repercussions' goes... Windows' share of the desktop OS has market has been declining for almost 20 years at the rate of about 1% per year. And about 70% of the global installed base is still on Windows 10.
They have a long way to fall, but I'm not sure that if I'm a regulator I look at that and say there needs to be some kind of intervention by society apart from what market forces are gradually doing anyway.
Windows development on eBPF is slower than Linux development on eBPF, so it will never be supported. A source code user licensee could develop it faster, but who licenses Windows source and already has great eBPF experience?
Microsoft already has an extensible file system filter capability in place, which is what current AV uses. Does it make sense to add eBPF on top of that and if so, are there any performance downsides, like we see with file system filters?
They've done a technology transition once already from legacy file system filter drivers to the minifilter model. If they see enough benefit to another change, it wouldn't be unprecedented.
Mind you, it looks like after 20-ish years Windows still supports loading legacy filter drivers. Given the considerable work that goes into getting even a simple filesystem minifilter driver working reliably, it's safe to assume that we'd be looking at a similarly protracted transition period.
As to the performance, I don't think the raw infrastructure to support minifilters is the major performance hit. The work the drivers themselves end up doing tends to be the bigger hit in my experience.
Keep in mind they don't just allow any old code to execute in the kernel.
They do have rigorous tests (WHQL), it's just Crowdstrike decided that was too burdensome for their frequent updates, and decided to inject code from config files (thus bypassing the control).
Is there any evidence that the config files had arbitrary code in them? The only analysis I'd seen so far indicated a parsing error loading a viral signature database that was routinely updated, but in this case was full of garbage data.
Not rigorous enough to have detected this flaw in the kernel sensor, although effectively any bug in this situation (an AV driver) can brick a machine. I imagine WHQL isn't able to find every possible bug in a driver you submit to them, they're not your QA team.
Doubt it. Microsoft is clearly over Windows. They continue to produce it but every release feels like "Ugh, fine, since you are paying me a ton of money."
Internally, Microsoft is running more and more workloads on Linux and externally, I've had .Net team tell me more than once that Linux is preferred environment for .Net. SQL Server team continues to push hard for Linux compatibility with every release.
EDIT: Windows Desktop gets more love because they clearly see that as important market. I'm talking more Windows Server.
I had read previously from an unverified SQL Server engineer that the thing they wanted most (with Linux support) was proper containerization (from a developer perspective). Apparently containers on Windows just don't cut it (which is why nobody uses them in production). Take it with a grain of salt though.
I don't think they'd ever admit that filesystem performance was an issue (though we all know it is; NTFS is over 30 years old!).
It's my understanding, having done benchmarks on file access on Windows, that NTFS itself is not the problem. It's old, but the revision of the on-disk structure that we use today hails from Windows XP, and it's about on par in terms of feature parity (and backwards compatibility, given that I can still read native NT 3.51 volumes on Windows 11) with ext4.
A lot of the weirdly bad performance comes from all of the machinery that Windows wraps around file access for things like filter drivers. As long as you don't, say, indiscriminately follow every CreateFile() with a CloseHandle() and instead treat handle closure like garbage collection, you can actually eke out pretty good performance.
That all said, yeah, Windows containers are less than great for what I'd argue is one strikingly glaring flaw: Docker container images are built from smss.exe upward. That makes them not immediately portable between ntoskrnl.exe releases.
It's just easier for everyone involved (outside Windows GUI clicker admins) if it runs on Linux. Containerization is easier, configuration is easier and operating system is much more robust.
Operating system can be more robust, depending on admin skill. Let idiots configure and operate your rhel and you may not get those five nines.
There are costs to it, in the form of architectural baggage and slower iteration, but what windows brings to the table is a deck swept mostly clear of footguns. That can give you a different form of robustness.
They aren't over windows. They continue to be incredibly interested in and actively developing how much money they can suck from their users. Especially via various forms of ads.
But yeah, kernel features are few and far between.
I believe the term you are looking for is "rent seeking". Other than visual changes, what new functionality does Windows 11 actually have that Windows XP didn't have? (I'm being generous with XP, because actually 95 was already mostly internet ready.) Yet how many times have many of us paid for a Windows license on a new computer or because the old version stopped getting updates?
> Other than visual changes, what new functionality does Windows 11 actually have that Windows XP didn't have?
Off the top of my head, limiting myself to just NT kernel stuff: WSL and Hyper-V, pseudo-terminals, condvars, WDDM, DWM, elevated privilege programs on the same desktop, font driver isolation, and limiting access to win32k for sandboxing.
> what new functionality does Windows 11 actually have that Windows XP didn't have? (
Off the top of my head, built-in bluetooth support, an OS-level volume mixer, and more support for a wider variety of class-compliant devices. I'm sure there are a lot more, and if you actually care about the answer, I don't think it would be hard to find.
Simple patches/upgrades vs tricking people into thinking you've made a whole new piece of software. Linux, BSD, and Apple roll out OS upgrades with new functionality without charging for the new versions.
That's one perspective I suppose. I have a MacBook on my desk at work solely for testing in Safari. I can no longer use it for that purpose because it won't even let me upgrade the OS. That sounds like a whole new piece of software to me. Windows actually has been substantially re-written. I guess MacOS has also? It seems more honest to me call it a different product.
Longhorn was a significant rewrite, actually. The two big upheavals in windows history were: 2000, which essentially scrapped the 95 lineage in favour of NT; and Vista, which kicked a lot of 3rd-party crap out of the kernel and added a quality gate for drivers.
> Other than visual changes, what new functionality does Windows 11 actually have that Windows XP didn't have?
Modern crypto ciphersuites that aren't utterly broken? Your best options for symmetric crypto with XP are 3DES (officially retired by NIST as of this year) and RC4 (prohibited in TLS as of RFC 7465).
(And if you think 3DES isn't totally broken by itself, you're right... except for the part where the ciphersuite in question is in CBC mode and is vulnerable to BEAST. Thanks, mandated ciphersuites.)
> Other than visual changes, what new functionality does Windows 11 actually have that Windows XP didn't have?
XP->Vista alone brought a bunch of huge changes that massively improved security (UAC), capability (64 bit desktops), and future-proofing (UEFI) among many many other things.
Some helpful Wikipedia editors have answered this question in excessive detail, so I'm just going to link those for more info. Also I'm going to start with what XP changed from 2003 both because it makes a good comparison and I'd argue 2000/NT 5.0 is the root of the modern Windows era. Your next sentence after the quote implies you probably won't have a problem with that.
Obviously some of this will be "fluff" and that's up to your own personal definitions, but to act like there haven't been significant changes in every major revision is just nonsense.
Well that Windows 11 article is laughably short, considering it's a major version. But I appreciate you taking the time to compile all those links.
My point is the vast majority of this stuff is either "fluff" or cosmetic changes or random things that 99% of users don't use OR they are security and bug patches. HN users are not typical, so I'm sure some of the Windows updates are very important for people like us.
Maybe to Microsoft this is a significant rewrite: "The Calculator has been completely rewritten in C# and includes several new features." (Just picked at random.) Ok, but like why? Who cares? What was wrong with the last calculator? Absolutely nothing. Also who even uses Windows calculator instead of Excel or their phone? Was calculator rewritten to justify an FTE somewhere at Microsoft?
I'm not trying to troll, but I am trying to be contrarian. I honestly feel like a majority of desktop users don't really think too hard about their OS. None of the existing OSes should be significantly rewritten unless they are just completely flawed. Like say Apple decides to ditch the microkernel or Linux goes to Rust. Most people need stability and security, not new calculator features or different button shading. I'm singling out Microsoft for being the only one that rent seeks for superfluous changes. Apple is notoriously bad about wasting users time with constant updates for dumb stuff, but at least it's free, except for the cost of time while your computer slowly reboots and updates.
I hate to dispute with someone like Brendan Gregg, but I'm hoping vendors in this space take a more holistic approach to investigating the complete failure chain. I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure. It may be true, but if we don't do the analysis we could leave ourselves open to blindspots. There may also be plenty of alternative approaches that should be considered and appropriately discarded.
I think the part I specifically dispute is the only negative outcome is wasted CPU cycles. That's likely the case for the class of bug, but there are plenty of failure modes where a bad ruleset could badly brick a system and make it hard to recover.
That's not to say eBPF based security modules isn't the right choice for many vendors, just that let's understand what risks they do and do not avoid, and what part of the failure chain they particularly address.
Just because you have not been aware of the discussions on this topic that have been happening for years, doesn't mean that they haven't been happening. This isn't some new analysis formed 3 days after an incident, this is the generally accepted consensus among many experts who have been working in the space, introducing these new APIs specifically to improve stability, security, etc. of systems.
> I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure.
Microsoft has been working on eBPF for a few years at least.
If you're really concerned, they have discussions and communication channels where you're invited to air your concerns. They're listed on their github:
This isn't right. If I need a system to run with a piece of code, then it shouldn't run at all if that piece of code is broken. Ignoring the failure is perverse. Let's say that the driver code ensures that some medical machine has safety locks (safeguards) in place to make sure that piece of equipment won't fry you to a crisp; I'd prefer that the whole thing not run at all rather than blithely operate with the safeguards disabled.
It's turtles all the way down.
I think the premise is false? It's up to the eBPF implementor what to do in the case of invalid input; the kernel could choose to perform a controlled shutdown in that case. (I have no idea what e.g. Linux actually does here, but one could imagine worlds where the action it takes on invalid input is configurable.)
Also your statement is sometimes not true, although I certainly sympathise in the mainline case. In some contexts you really do need to keep on trucking. The first example to spring to mind is "the guidance computers on an automated Mars lander"; the round-trip to Earth is simply too long to defer responsibility in that case. If you shut down then you will crash, but if you do your best from a corrupted state then you merely probably crash, which is presumably better.
> I have no idea what e.g. Linux actually does here
If you attempt to load an eBPF program that the verifier rejects, the syscall to load it fails with EINVAL or E2BIG. What your user-space program then does is up to you, of course.
The medical machine software should just refuse to run with an error message if a critical driver was not loaded.
The OS bricking is causing way more trouble where an IT technician now needs to fix something where it otherwise would just be updating the faulty driver...
Also does your car not start if you are missing water for the wiper?
3rd party hooking into kernel is 3rd party responsibility. It is like equipping your car with LPG - THAT hooks into engine (kernel). And When I had a faulty gas pressure sensor then my car actually halted (BSOD if you will) instead of automatically failing over to gasoline as it is by design.
You can argue that car had no means to continue execution but kernel has, however invalid kernel state can cause more corruption down the road. Or as parent even points out - carry out lethal doses of something.
Initially I was inclined to disagree ("these things should always fail safe") however with more and more stuff being pushed into the kernel it's hard to say that you're wrong or exactly where a line needs to be drawn between "minimally functional system" and "dangerously out of control system".
I think until we discover a technology that forces commercial software vendors to employ functioning QA departments none of this will really solve anything.
I agree that some system components should be treated as critical no matter what, but the software at issue in this case (Falcon Sensor or Antivirus more generally) is precautionary and only best effort anyways. I would wager the vast majority of the orgs affected on Friday would have preferred the marginally increased risk of a malware attack or unauthorized use over a 24 hour period instead of the total IT collapse they experienced. Further, there's no reason the bug HAD to cause a BSOD, it's possible the systems could have kept on trucking but with an undefined state and limitless consequences. At least with eBPF you get to detect a subset of possible errors and make a risk management decision based on the result.
I'm with you.
What's critical, and what's not? Is it a big thing, or not a big thing? Is this particular machine more critical than the one over there?
Security systems need to be at the lowest level, or else some shifty bastard will find a path around them. If it's at the lowest level, the downside of a failure is catastrophic, as we experienced last Friday. The carnage here is ultimately on CrowdStrike. The testing must have been slapdash at best, and missing at worst. eBPF changes nothing. The question is: should we fail, or carry on? eBPF doesn't help with that decision, it only determines the outcome from a system perspective. Any decision is a value judgement; it might be right or wrong, and its outcome either benign or deadly.
Choices!
I like how Unison works for this reason. You call functions by cryptographic hash, so you have some assurance that you're calling the same function you called yesterday.
Updates would require the caller to call different functions which means putting the responsibility in the hands of the caller, where it should be, instead of on whoever has a side channel to tamper with the kernel.
You end up with the work-perfectly-or-not-at-all behavior that you're after because if the function that goes with the indicated hash is not present, you can't call it, and if it is present you can't call it in any way besides how it was intended
The system clearly already behaves that way (i.e. ignores failure) - after all, the fix was to simply delete the offending file. If that's an option, then loader can do that too. It can and perhaps even is smarter, such as "fallback onto previous version".
Furthermore, the reaction to a malformed state need not be "ignore". It could disable restricted user login; or turn off the screen.
If the worry is that this is viable to abuse by malware, well, if the malware can already rewrite the on-disk files for the AV, I wonder whether it's really a good idea to trust the system itself to be able to deal with that. It'd probably be safer to just report that up the security foodchain, and potentially let some external system take measures such as disable or restrict network access. Better yet, such measures don't even require the same capabilities to intervene in the system, merely to observe - which makes the AV system less likely to serve as a malware vector itself or to cause bugs like this.
If the failed system is a security module, I think that's absolutely correct. If the system runs, without the security module, well, that's like forgetting to pack condoms on Shore Leave. You'll likely be bringing something back to the ship with you.
Someone needs to be testing the module, and the enclosing system, to make sure it doesn't cause problems.
I suspect that it got a great deal of automated unit testing, but maybe not so much fuzz and monkey (especially "Chaos Monkey"-style) testing.
Interesting analogy, but yes. If the module *is* necessary, well, it's necessary and nothing should work without it. Testing must have been a mess here.
> In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code.
eBPF is fantastic, and it can be used for many purposes and improve a lot of things, but this is IMO overselling it. Assuming that BPF itself it free of bugs, it’s still a rather large sprawl of kernel hooks, and those hooks invoke eBPF code, which can call right back into the kernel. Here’s a list:
bpf_probe_read_kernel() is particularly heavily used, and it is not safe. It tries fairly hard not to OOPS or crash, but it is definitely not perfect.
The rest of that list contains plenty of this that will easily take down a system, even if it doesn’t actually oops or panic in the process.
And, of course, any tool that detects userspace “malicious behavior” and stops it can start calling everything malicious, and the computer becomes unusable.
Meanwhile, eBPF has no real security model on the userspace side. Actual attachment of an eBPF program goes through the bpf() syscall, not through sensibly permissioned operations on the underlying kernel objects being attached to, and there is nothing whatsoever that confines eBPF to, say, a container that uses it. (See bpf_probe_read_kernel() -- it's fundamentally able to read all kernel memory.)
So, IMO, most of the benefit of eBPF over ordinary kernel C code is that eBPF is kind of like writing code in a safe language with a limited unsafe API surface. It's a huge improvement for this sort of work, but it is not perfect by any means.
> The verifier is rigorous -- the Linux implementation has over 20,000 lines of code
The verifier is absurdly complex. I'd rather see something based on formal methods than 20kLOC of hand-written logic.
I'm not sure that "panic" is the right word here. bpf_probe_read_kernel boils down to copy_from_kernel_nofault, which checks for an "allowed" address and then does the access. Any page faults turn into error returns instead of OOPSes. x86 disallows user addresses, the vsyscall page, and non canonical addresses.
Doing this from bpf assumes that all "allowed" addresses are side-effect-free and will either succeed or cleanly fault. Off the top of my head, MMIO space (including, oddities like the APIC page on CPUs that still have that) and TDX memory are not in this category.
> eBPF programs cannot crash the entire system because they are safety-checked by a software verifier and are effectively run in a sandbox.
Isn’t one of the purposes of an OS to police software? I get that this has to do with the OS itself, but what does watching the watchers accomplish other than adding a layer which must then be watched?
Why not reduce complexity instead of naively trusting that the new complexity will be better long term?
eBPF isn't "watching the watchers" it's just a tool that lets other tools access low-level things in the kernel via a very picky sandbox. Think of it like this:
Old way: Load kernel driver, hook into bazillions of system calls (doing whatever it is you want to do), pray you don't screw anything up (otherwise you can get a panic though not necessarily--Linux is quite robust).
eBPF way: Just ask eBPF to tell you what you want by giving it some eBPF-specific instructions.
Right? I might spend a few minutes seeing if an AI chatbot can explain all the justifications that lead to using something like CrowdStrike in the first place.
This sounds like a cool technology, but this was the really egregious problem:
> There are other ways to reduce risks during software deployment that can be employed as well: canary testing, staged rollouts, and "resilience engineering" in general
You don't need a new technology to implement basic industry-standard quality control
Maybe we should start taking Fridays off to commemorate the event, which probably would have been less bad if more people spent less time with their nose to the grindstone and had more time to stop and think about how it all was shaping up and how they could influence that shape.
```The verifier is rigorous -- the Linux implementation has over 20,000 lines of code -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). The safety this provides is a key benefit of eBPF, along with heightened security and lower resource usage.
```
Wow, 20k is not exactly encouraging. Besides the extra attack surface, who can vouch for such a large code base?
I had exactly the same thought. I don’t know if that 20k number was supposed to inspire confidence, but for me it did the opposite. It would have inspired confidence if it was 300 lines of code.
My impression is that the WebAssembly verifier is much simpler.
If the filters are loaded at boot and hook into everything then a bug can still lock down the system to a point where it can't be operated or patched anymore (e.g. because you loaded an empty whitelist). So it could end up replacing a boot loop with another form of DoS.
If microsoft includes a hardcoded whitelist that covers some essentials needed for recovery that could make a bug in such a tool easier to fix, but could still cause effective downtimes (system running but unusuable) until such a fix is delivered.
I tried to Google about this, but I cannot find anything definitive. It looks like you can still break things. Can an expert on eBPF please comment on this claim? This is the best that I could find: https://stackoverflow.com/questions/70403212/why-is-ebpf-sai...
eBPF programs cannot crash the kernel, assuming there are no bugs in the eBPF verifier. There have been such bugs in the past but they seem to be getting more and more rare.
I don't see how this contradicts what I said. Indeed, there are helpers, but the verifier is supposed to check that the eBPF program isn't calling them with invalid arguments.
Yes, bugs in Linux are possible, so there might be some eBPF code that crashes the kernel. Just like bugs in Chrome are possible, so there might be some JavaScript that crashes the browser. Still, JavaScript is much safer than native code, because fixing the bugs in one implementation is a tractable problem, whereas fixing the bugs in all user code is not.
"These security agents will then be safe and unable to cause a Windows kernel crash."
Unless of course there is a bug in eBPF (https://access.redhat.com/solutions/7068083) @brendangregg and the kernel panics/ BSoDs anyway which you mention later in the article of course.
This is true but the kernel gets more scrutiny and has better priorities. Only CrowdStrike audits and hardens the CS kernel driver, so things like proactive improvements are competing in a single Jira board against marketing’s request for new features (want to bet that was all AI until Friday?) whereas the kernel eBPF implementation might be improved by people at other security vendors, distributions like Red Hat or Ubuntu or a major cloud provider (all of whom fund serious security audits and have engineers who care a lot about robustness), or academic researchers.
“Many eyes” is a bit dubious in general but the Linux kernel is pretty much the best case for it being true.
> In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code.
Assuming every security critical system will be on a recent enough kernel to support this...
$ cat /etc/redhat-release /etc/oracle-release /proc/version
Red Hat Enterprise Linux release 8.10 (Ootpa)
Oracle Linux Server release 8.10
Linux version 5.15.0-203.146.5.1.el8uek.x86_64 (mockbuild@host-100-100-224-48) (gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9.2.0.1), GNU ld version 2.36.1-4.0.1.el8_6) #2 SMP Thu Feb 8 17:14:39 PST 2024
Considering the number of systems running very obsolete OSes these days: WinNT (4x or 3x), Windows, DOS, or various proprietary Unixen, stale Linux flavours, etc., etc., ... yes, quite.
RHEL kernel.. right. Imho, I'd trust an upstream stable kernel far more than a RHEL one for production which has dozen of feature backports and an internal kABI to maintain.. granted RH has a QA team, but it is still impossible to test everything beforehand.
Ok. But the good old push code to staging / canary it before mainstream updates was a simpler way of solving the same problem.
Crowdstrike knows the computers they're running on, it is trivial to implement a system where only few designated computers download and install the update and report metrics before the update controller decides to push it to next set.
It would mitigate the problem, but not solve it. You can still imagine a condition that only occurs after the update has been rolled out everywhere. Furthermore, such a bug would still be extremely problematic for the concerned customers, even if not all of them were affected. In addition, it would be necessary to react very quickly in the case of zero-day vulnerabilities.
Yes, I am not arguing against having the ability to deal with it quickly - I am saying canary/ staging helps you do exactly that. Because as we see in the case of Intel CPUs and Crowdstrike some problems or scale of some problems is best prevented.
"Mitigation" is dealing with an outage/breakage after it occurs, to reduce the impact or get system healthy again.
You're talking about "prevention" which keeps it from happening at all.
Canarying is generic approach to prevention, and should not be skipped.
Avoiding the risk entirely (eBPF) would also help prevent outage, but I think we're deluding ourselves to say it "solves" the problem once and for all; systems will still go down due to bad deploys.
with the way they handled the debian crashing a little while ago, frankly they are happy to still go ahead with testing this way. still much better way to handle things than pushing to everybody at the same time.
Why trust somebody else not messing up? With that in place for windows and crowdstrike billions of dollars would be saved and many lives not negatively impacted
...
The implicit assumption of the article is that eBPF code can't crash a kernel, but the article itself eventually admits that it can and has done, including last month. eBPF is a safer way of providing kernel-extension functionality, for sure, but presenting it as the perfect solution is just asking to have your argument dismissed. eBPF is not perfect. And there's plenty of things it can't do. The very sandbox rules that limit how long its programs may run and what they can do also make it entirely inappropriate for certain tasks. Let's please stop pretending there's a silver bullet.
It's not a silver bullet, however, it is still better to pushing all the panicable bugs into one community-maintained section (e.g. eBPF verifier). All vendors have an incentive to help get right and this is much better than every vendor shipping their own panicable bugs in their own out of tree kernel modules. Additionally, it's not just the industry looking at eBPF, but also academia in terms of formally verifying these critical sections.
It's casually claiming to have solved the halting problem, at least within some limited but useful context. That should be impossible, and it turns out, it is.
I expect it can be solved within some limited contexts, but those contexts are not useful, at least not at the level of "generic kernel code".
It solves the halting problem by not being Turing complete. I presume each eBPF runs in a context with bounded memory, requested up front, for one thing; it also disallows jumps unless you can prove the code still halts.
eBPF started out as Berkeley Packet Filters. People wanted to be able to set up complex packet filters. Things like 'udp and src host 192.168.0.3 and udp[4:2]=0x0034 and udp[8:2]=0x0000 and udp[12]=0x01 and udp[18:2]=0x0001 and not src port 3956'
So BPF introduced a very limited bytecode, which is complex enough that it can express long filters with lots of and/or/brackets - but which is limited enough it's easy to check the program terminates and is crash-free. It's still quite limited - prior to ~2019, all loops had to be fully unrolled at compile time as the checker didn't support loops.
It turned out that, although limited, this worked pretty well for filtering packets - so later, when people wanted a way to filter all system calls they realised they could extend the battle-tested BPF system.
Nobody is claiming to have solved the halting problem.
Did you read the article? It says computers will not crash in the future due to updates. It literally says that in the very first line of the article.
> In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code.
What you are claiming is completely different. A kind of "firewall" for syscalls. But updates to drivers and software must contain code and data. The author is not talking about updates to the firewall between drivers and the kernel, they talk about updating drivers themselves. It literally says "updates that involve kernel code". Will the kernel only consist of eBPF filtering bytecode? How could that possibly work?
"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair."
I don't do any kernel stuff so I'm out of my element, but doesn't the fact that Crowdstrike & Linux kernel eBPF already caused kernel crashes[1] sort of downplay the rosiness of the state of things?
Can you elaborate? What I see about Linux is that Crowdstrike was in the process of adopting eBPF which is ostensibly immune to kernel panics, but that issue shows their eBPF implementation specifically causing a kernel panic.
I've read it three times now. The only thing they say about it is this:
"This doesn't mean that eBPF has solved nothing, substituting a vendor's bug for its own. Fixing these bugs in eBPF means fixing these bugs for all eBPF vendors, and more quickly improving the security of everyone."
Which is exactly what I'm asking about. If eBPF has some inherent advantage, why did it fail in precisely the same way alreay?
Lets suppose that eBPF solves this particular problem, eventually, for Windows. Doesn't sidestepping the entire class of Crowdstrike-style fubars require that Microsoft then mandate that no, backward compatibility will not be offered?
Back compat seems to be such a shibboleth in the Windows world, but comes at an incredible price. The reasons cited all seem to boil down to keeping some imagined customers' obscure LOB app running for decades. But that seems like an excuse to me. Surely Microsoft would like to shake out the last diehards running some VB5 app on a patched up PC in a factory. Isn't it more beneficial to everyone to start sunsetting acres of ancient NT code and approaches and streamline the entire attack surface?
Backwards compatibility slows things down in the Windows world but it doesn’t halt improvements. In this case, there are two powerful ratchets:
1. Compliance: everyone affected by this bug has auditors. Once safer alternatives are available, the standards like CIS, PCI, etc. will be updated to say you should use the new interface, and every enterprise IT department will have pressure to switch to eBPF tools. We saw this with BootLocker: storage encryption used to be a pain, people resisted it, but over time it became universal because the cost of swimming upstream was too high.
2. Signing. Microsoft can start requiring more proof of need and restrictions for signing drivers. They have to be careful to avoid the appearance of favoritism but after this debacle that’s a LOT easier. I would bet some engineer is working on a draft of mandatory fault handling and testing proof requirements for critical kernel drivers now and I would not be surprised to see it include a timeframe for adopting memory-safe languages.
>Surely Microsoft would like to shake out the last diehards running some VB5 app on a patched up PC in a factory.
>Isn't it more beneficial to everyone to start sunsetting acres of ancient NT code and approaches and streamline the entire attack surface?
If your code somehow still relies on some buggy behaviour to work, then MS shouldn't do anything to preserve that anymore - apparently they used to, but I'm not so sure nowadays.
However 'ancient NT' code should probably still function just fine since the Win32 API hasn't changed much for a while, and MS don't actively deprecate function calls (unlike Apple who seem to do it a bit on a whim recently). I would put this down to the API being pretty well designed in the first place.
So many problems though! including commercial monocultures, lack of update consent, blast radius issues, etc etc. There's a commons in our pockets but that is very difficult to regulate for. The will keep putting the gun to your head until you keep choosing the monoculture.
WebAssembly is a better choice for sandboxing kernel code. It has a full formal specification with a mechanized proof of type safety, many high-performance implementations, broad toolchain support, is targetable from many languages, and a capability security model.
Hardly. For starters, wasm doesn’t guarantee that a piece of code terminates in bound time. There are further security guarantees in ebpf such as any lock acquired must be released.
You can apply additional static checks to Wasm, e.g. control flow analysis, and reject programs without obvious loop bounds or unbalanced locking operations. Or you could apply dynamic techniques like tracking acquired locks and automatically releasing them, or charging fuel (gas). The latter is quite common for blockchain runtimes based on Wasm.
> The worst thing an eBPF program can do is to merely consume more resources than is desirable, such as CPU cycles and memory.
This is obviously not true. It might be the worst it can do, by itself, to the currently running kernel. It's not the worst it can do to the machine or its user(s).
There are infinite harmful things an eBPF program can do. As can programs solely in user-space. There is a specific class of vulnerabilities being mitigated by moving code from kernel to BPF. That does not mean that eBPF programs are in general safe.
Does anyone know how far along the eBPF implementation for Windows actually is? In the sense that it could start feasibly replacing existing kernel drivers.
Even if Microsoft rolls out eBPF and mainstreams it - it will be years before everything is ported over and it still won't address legacy windows versions (which appear to be a good chunk of what was impacted).
It's a move in the right direction but it probably won't fully mitigate issues like this for another 5+ years.
Sure, but 5 years is not that long ago - for example, if they’d started right before the pandemic it’d be almost done by now. The best time to have done that was 5 years ago but the second best time is now.
> an unprecedented example of the inherent dangers of kernel programming
I take issue with that. Kernel programming was not to blame; looking up addresses from a file and accessing those memory locations without any validation is. The same technique would yield the same result at any Ring.
_No_ difference oversells it, IMO -- the fact that the entire OS crashed is what made fixing the bug so arduous, since it required in-person intervention. To be sure, running the code in userspace would still cause unacceptable service interruptions, but the fix could be applied remotely.
I agree for some systems. For others, stopping the system has bigger consequences than not having cyber protection for a few hours because of a bug that’ll get fixed. For example, hospitals, or possibly Delta Airlines.
Can someone tell me what's the advantage of eBPF over a user mode driver? The article makes it look it eBPF is have your cake and eat it too solution which is too good to be true? Can you run graphics drivers in eBPF for example?
AFAIK, an ebpf function can only access memory it got handed as an argument or as result from a very limited number of kernel functions. Your function will not load if you don't have boundary checks. Fighting the ebpf validator is a bit like fighting Rust's borrow checker; annoying, at times it's too conservative and rejects perfectly correct code, but it will protect you from panics. Loops will only be accepted if the validator can prove they'll end in time; this means it can be a pain to make the validator to accept a loop. Also, ebpf is a processor-independent byte code, so vectorizing code is not possible (unless the byte code interpreter itself does it).
Given all its restrictions, I doubt something complex like a graphics driver would be possible. But then, I know nothing about graphics driver programming.
> Fighting the ebpf validator is a bit like fighting Rust's borrow checker
I think this undersells how annoying it is. There's a bit of an impedance mismatch. Typically you write code in C and compile it with clang to eBPF bytecode, which is then checked by the kernel's eBPF verifier. But in some cases clang is smart enough to optimize away bounds checks, but the eBPF verifier isn't smart enough to realize the bound checks aren't needed. This requires manual hacking to trick clang into not optimizing things in a way that will confuse the verifier, and sometimes you just can't get the C code to work and need to write things in eBPF bytecode by hand using inline assembly. All of these problems are massively compounded if you need to support several different kernel versions. At least with the Rust borrow checker there is a clearly defined set of rules you can follow.
This is the wiki. I haven't kept up, but this isn't a kernel module.
"eBPF is a technology that can run programs in a privileged context such as the operating system kernel. It is the successor to the Berkeley Packet Filter (BPF, with the "e" originally meaning "extended") filtering mechanism in Linux and is also used in non-networking parts of the Linux kernel as well."
No, you can't run arbitrary general-purpose programs in eBPF, and you cannot run graphics drivers in it. You generally can't run programs with unprovably bounded loops in eBPF, and your program can interact with the kernel only through a small series of explicitly enumerated "helpers" (for any given type of eBPF program, you probably have about 20 of these in total).
1. How does eBPF solve this? It makes it more difficult, sure, but it'll almost always be possible to cause a crash, if you try hard enough.
2. More importantly, the problem is rarely fixable by changing technology, because typically, problems are caused by people and their connections: social/corporate pressures, profit-seeking, mental health being treated as unimportant, et cetera. eBPF can't fix those, and as long as corporations have social structures that penalize thoroughness and caution, and incentivize getting 'the most stuff' done, this will persist as a problem.
I don't buy it... didn't a bug from RedHat + Crowdstrike have a similar panic issue? I understand in that case it was because of RedHat, but still. I don't think this, by itself will change much.
Thanks! This was not a familiar acronym to me... and after some digging[0] apparently it's no longer an acronym:
"BPF originally stood for Berkeley Packet Filter, but now that eBPF (extended BPF) can do so much more than packet filtering, the acronym no longer makes sense. eBPF is now considered a standalone term that doesn’t stand for anything."
> If your company is paying for commercial software that includes kernel drivers or kernel modules, you can make eBPF a requirement. It's possible for Linux today, and Windows soon. While some vendors have already proactively adopted eBPF (thank you), others might need a little encouragement from their paying customers.
How about Microsoft's large government and commercial customers make it a requirement that MS does not develop a single new feature for the next two fucking years or however long it takes to go through the entirety of the Windows+Office+Exchange code base and to make sure there are no security issues in there?
We don't need ads in the start menu, we don't need telemetry, we don't need desktop Outlook becoming a rotten slow and useless web app, we don't need AI, we certainly don't need Recall. We need an OS environment that doesn't need a Patch Tuesday where we have to check if the update doesn't break half the canary machines.
And while MS is at that they can also take the goddamn time and rework the entire configuration stack. I swear to god, it drives me nuts. There's stuff that's only accessible via the registry (and there is no comprehensive documentation showing exactly what any key in the registry can do - large parts of that are MS-internal!), there's stuff only accessible via GPO, there's stuff hidden in CPLs dating back to Windows 3.11, and there's stuff in Windows' newest UI/settings framework.
Here's an idea for an interesting hack: a piece of kernel resident code that feeds fake data into eBPF so that an eBPF-based antimalware will see nothing bad as the malware goes about it's merry way.
Sandboxes are safe, but are ultimately virtual machines, and virtual machines can be made to live in a world that's not real.
I don’t understand statements like this. You only need to have some employee install some malware (unintentionally or otherwise); and you have a data breach on your hands.
> If your company is paying for commercial software that includes kernel drivers or kernel modules, you can make eBPF a requirement.
Windows soon, may still be atleast a year ahead. Would that be a fair statement? atleast being the operating keyword here.
Specifically in the context of network security software, for eBPF programs to be portable across windows/linux, we would need MSFT to add a lot more hooks and expose internal kernel stucts. Hopefully via a common libbpf definition.
Otherwise, I fear, having two versions of the same product, across two OSs would mean more secuirty and quality issues.
I guess the point I am trying to make is, we would get there, but we are more than a few years away.
I would love to see something like cilium on vanilla windows for a Software defined Company Wide network. We can then start building enterprise network secutiry into it. Baby steps!
---
btw, your talks and blog posts about bpftools is godsent!
Because it used to stand for extended Berkeley Packet Filter and it has since moved far, far beyond just packets. It now hooks into the entire network stack, security, and does observability/tracing for nearly anything and everything in the kernel ("nearly" because some stuff runs when the kernel boots up--before eBPF is loaded--and never again after that).
In the future, computers will not crash due to bad software updates, even those updates that involve kernel code.
100% BS. Even if they don't "crash" they will "stop functioning as intended" which is just the same. It's absolutely disgusting how this industry is now using this one outage as a talking point to further their totalitarian agenda.
It reminds me of how Google went after adblockers with their new extension model that also promised more "security". It's time we realised what they're really trying to do. In fact, I wonder whether this outage was not accidental after all.
If you consider kernel programming to be inherently unsafe, then you would consider this to be inevitable, meaning it's not really the specific company's fault. They were just the unlucky ones.
Right, and we wanted to talk about all security solutions and not make this about one company. We also wanted to avoid shaming since they have been seriously working on eBPF adoption, so in that regard they are at the forefront of doing the right thing.
They could have helped their luck by doing some of the common sense things suggested in the article.
For instance, why not find a subset of your customers that are low risk, push it out to them, and see what happens? Or perhaps have your own fleet of example installations to run things on first. None of which depends on any specific technology.
> If I were the customers and I found out that I was used as test subject, how would I feel?
In reality, every business has relationships that it values more than others. If I wasn't paying a lot for it, and if I was running something that wasn't critical (like my side project) then why not? You can price according to what level of service you want to provide.
Canary deployment for subset of Salesforce customers won't see much of revolt from customers compare to AV definition rollout (not software, but AV definition) in Cybersecurity where gaps between 0day and rollout means you're exposed.
If customers found out that some are getting roll out faster than the others, essentially splitting the group into 2, there will be a need for customer opt-in/opt-out.
If everyone is opting-out because of Friday, your Canary deployment becomes meaningless.
Any proof that other Cybersecurity vendors do Canary deployment for their AV definition? :)
PS: not to say that the company should test more internally...
Canary deployment doesn’t necessarily mean massive gaps between deployment waves. You can fast-follow. Sure, there may be scenarios with especially severe vulnerabilities where time is of the essence. I’m out of the loop if this crowdstrike update was such a scenario where best practices for software deployment were worth bypassing.
If this is just how they roll with regular definition updates, then their deployment practices are garbage and this kind of large scale disaster was inevitable.
Let's walk this through: Canary deployment to Windows machines. If those Windows machines got hit with BSOD, they will go offline. How do you determine if they go offline because of Canary or because of regular maintenance by the customer's IT cycle?
You can guess, but you cannot be 100% sure.
What if the targeted canary deployments are Employees desktops that are OFFLINE during the time of rollout?
>I’m out of the loop if this crowdstrike update was such a scenario where best practices for software deployment were worth bypassing.
I did post a question: what about other Cybersecurity vendors? Do you think they do canary deployment on their AV definitions?
Cybersecurity companies participate in Sec evaluation annually that evaluates (measure) and grade their performance. That grade is an input for Organizations to select vendors outside their own metrics/measurements.
I don't know if MTTD is included in the contract/SLA. If it does, you got some answer as to why certain decision is made.
It's definitely interesting to see Software developers of HN giving out their 2c for a niche Cybersecurity industry.
I worked in the cyber security space for a decent chunk of my career, and the most frustrating part was cyber security engineers thinking their problems were unique and being completely unaware of the lessons software engineering teams have already learned.
Yes, you need to tune your canary deployment groups to be large and diverse enough to give a reliable indicator of deployment failure, while still keeping them small enough that they achieve their purpose of limiting blast radius.
Again, if you follow industry best practices for software deployment, this is already something that should be considered. This is a relatively solved problem -- this is not new.
> I did post a question: what about other Cybersecurity vendors? Do you think they do canary deployment on their AV definitions?
I think that question is being asked right now by every company using Crowdstrike — what vendors are actually doing proper release engineering and how fast can we switch to them so that this never happens to us again?
>if you follow industry best practices for software deployment, this is already something that should be considered. This is a relatively solved problem -- this is not new.
You have to ask the customer if they're okay with that citing "our software might failed and brick your machine".
I'd like to see any Sales and Marketing folks say that ;)
> I think that question is being asked right now by every company using Crowdstrike — what vendors are actually doing proper release engineering and how fast can we switch to them so that this never happens to us again?
Uber valid question and this BSOD incident might be a turning point for customers to pay up more for their IT infrastructure.
It's like: previously Cybersecurity vendors are shy to ask customers to setup Canary systems because that's just "one-more-thing-to-do". After BSOD: customers will smarten up and do it without being asked and to the point where they would ask Vendors to _support_ that type of deployment (unless they continue to be cheap and lazy).
> You have to ask the customer if they're okay with that citing "our software might failed and brick your machine".
I think you’re still missing the point of Canary deployments. The question your sales team should ask is “would you like a 5% chance of a bug harming your system, or a 100% chance?”
> It's like: previously Cybersecurity vendors are shy to ask customers to setup Canary systems because that's just "one-more-thing-to-do"
You should by shy because it is not your customer’s job to set up canary deployments. Crowdstrike owns the software and the deployment process. They should be deploying to a subset of machines, measuring the results, and deciding whether to roll forward or roll back. It is not the customers job to implement good release engineering controls for Crowdstrike (although after this debacle you may well see customers try).
If you refer Canary deployment as the vendor's internal deployment? I definitely agree.
What I find it hard is those in Software that suggested to roll it to a few customers first because this isn't cloud deployment doing A/B test when it comes to Virus Definition.
Customers must know what's going on when it comes to virus definition and the implication of them whether they're being part of the rollout group or not.
> If you refer Canary deployment as the vendor's internal deployment? I definitely agree.
No, I’m talking about external deployment to customers. They clearly also had a massive failure in their internal processes too, since a bug this egregious should never make it to the release stage. But that is not what I am talking about right now.
> What I find it hard is those in Software that suggested to roll it to a few customers first because this isn't cloud deployment doing A/B test when it comes to Virus Definition.
I don’t care what you’re releasing to customers— application binary, configuration change, virus definition, etc, if it has the chance of doing this much damage it must be deployed in a controlled, phased way. You cannot 100% one-shot deploy any change that has the potential to boot-loop a massive amount of systems like this. This current process is unacceptable.
> Customers must know what's going on when it comes to virus definition and the implication of them whether they're being part of the rollout group or not.
Who says they don’t have to know? Telling your customers that an update is planned and giving them a time window for their update seems reasonable to me.
Why even do that? We have virtualization, they could emulate real clients and networks of clients. This particular bug would have been prevented for sure
Yeah I thought maybe the VM thing might not catch the bug for some reason, but it seems like the natural thing to do. Spin up VM, see if there's a crash. I heard the technical reason had something to do with a file being full of nulls, but that sort of thing you should catch.
Honestly, the most generous excuse I can think of is that CS were informed of some sort of vulnerability that would have profound consequences immediately, and that necessitated a YOLO push. But even that doesn't seem too likely.
Agree, Crowdstrike was an unlucky one, but it is more about the issue in general. If I remember correctly, also others like sysdig user their own kernel modules for collection.
I still hold true that testing even improperly would have caught this before it hit worldwide. But I suppose you are right, that doesn’t help the argument being made here.
CrowdStrike is mentioned, but the goal of the article is to promote eBPF. CrowdStrike is tangentially related because it draws attention to a platform that Gregg has put a lot into.
But the appeal-to-authority evidence that the article presents is not.
"-- the Linux implementation has over 20,000 lines of code -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). The safety this provides is a key benefit of eBPF, along with heightened security and lower resource usage."
Sorry, but neither eBPF nor Rust nor formal verification nor ... is going to solve that problem. Repeat after me: there are no technical solutions to social problems. As long as the result of such an outage is basically a "oh, a software problem! shrug", _nothing_ will change.
I wonder if microkernels ever had this kind of bullshit. Had it been a microkernel, would we all be sitting twiddling our thumbs on friday? Hot take: No.
> If the verifier finds any unsafe code, the program is rejected and not executed. The verifier is rigorous -- the Linux implementation has over 20,000 lines of code [0] -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington).
/* bpf_check() is a static code analyzer that walks eBPF program
* instruction by instruction and updates register/stack state.
* All paths of conditional branches are analyzed until 'bpf_exit' insn.
*
* The first pass is depth-first-search to check that the program is a DAG.
* It rejects the following programs:
* - larger than BPF_MAXINSNS insns
* - if loop is present (detected via back-edge)
...
I haven't inspected the code, but I thought that checking for infinite loops would imply solving the halting problem. Where's the catch?
I'm not able to comment on what this code is doing, but as for the theory:
The halting problem is only unsolvable in the general case. You cannot prove that any arbitrary piece of code will stop, but you can prove that specific types of code will stop and reject anything that you're unable to prove. The trivial case is "no jumps"—if your code executes strictly linearly and is itself finite then you know it will terminate. More advanced cases can also be proven, like a loop over a very specific bound, as long as you can place constraints on how the code can be structured.
As an example, take a look at Dafny, which places a lot of restrictions on loops [0], only allowing the subset that it can effectively analyze.
Adding on (and it's not terribly relevant to eBPF), it's also worth noting that there are trivial programs you can prove DON'T halt.
A trivial example[1]:
int main() {
while (true) {}
int x = foo();
return x;
}
This program trivially runs forever[2], and indeed many static code analyzers will point out that everything after the `while (true) {}` line is unreachable.
I feel like the halting problem is incredibly widely misunderstood to be similar to be about "ANY program" when it really talks about "ALL programs".
[1]: In C++, this is undefined behavior technically, but C and most other programming languages define the behavior of this (or equivalent) function.
The halting problem cannot be solved in the general case, but in many cases you can prove that a program halts. eBPF only allows verifiably-halting programs to run.
the halting problem is only true for _arbitrary_ programs
but there are always sets of programs for which it is clearly possible to guarantee their termination
e.g. the program `return 1+1;` is guaranteed to halt
e.g. given program like `while condition(&mut state) { ... }` with where `condition()` is guaranteed to halt but otherwise unknown is not guaranteed to halt, but if you turn it into `for _ in 0..1000 { if !condition(&mut state) { break; } ... }` then it is guaranteed to halt after at most 1000 iterations
or in other words eBPF only accepts programs which it can proof will halt in at most maxins "instruction" (through it's more strict then my example, i.e. you would need to unroll the for-loop to make it pass validation)
the thing with programs which are provable halting is that they tend to also not be very convenient to write and/or quite limited in what you can do with them, i.e. they are not suitable as general purpose programming languages at all
This, others have said it less concisely, but a program without loops and arbitrary jumps is guaranteed to halt if we assume the external functions it calls into will halt.
I'm glad to hear that Meta and Google code is "rigorous". I'd prefer INRIA, universities that fund theorem provers, industries where correctness matters like aerospace or semiconductors.
Windows doesn't use the Linux eBPF verifier, they have their own implementation named PREVAIL[0] that is based on an abstract interpretation model that has formal small step semantics. The actual implementation isn't formally proven, however.
The halting problem is exhaustive, there isn't an algorithm that is valid for all programs. You can still check for some kinds of infinite loops though!
More specifically, you can accept a set of programs that you are certain do halt, and reject all others, at the expense of rejecting some that will halt. As long as that set is large enough to be practical, the result can be useful. If you eg forbid code paths that jump "backwards", you can't really loop at all. Or require loops to be bounded by constants.
I have no insight into this particular project but you could work around the halting problem by only allowing loops you can proof will not go infinite. That would of course imply rejecting loops that won't go infinite but can't be proven not to.
If the verifier can't determine that the loop will halt, the program is disallowed. Also, if the program gets passed and then runs too long anyway, it's force-halted. So... I guess that solves the halting problem.
So this "solves" the halting problem by creating a new class "might-not-halt-but-not-sure" and lumping it with "does-not-halt". I find it hard to believe the new class is small enough for this to be useful, in the sense that it will avoid all kernel crashes.
I rather expect useful or needed code would be rejected due to "not-sure-it-halts", and then people will use some kind of exception or not use the verifier at all, and then we are back to square one.
Well it is useful in practice, there are some pretty useful products based on eBPF on Linux, most notably Cilium (and, shameless plug for the one I’m working on: Parca, an eBPF-based CPU profiler).
Bad wording on my part, and I still don't know how to word it better. I'm sure this thing is useful, I don't think everyone who contributed code was just clueless.
However, the claim "in the future, computers will not crash due to bad software updates, even those updates that involve kernel code" must be false. There is no way it is true. Whatever Cilium is, I cannot believe it generally prevents kernel crashes.
Correct, you will never be able to write any possible arbitrary code and have it run in eBPF. It necessarily constrains the class of programs you can write. But the constrained set is still quite useful and probably includes the crowdstrike agent.
Also, although this isn't the case now, it's possible to imagine that the verifier could be relaxed to allow a Turing-complete subset of C that supports infinite loops while still rejecting sources of UB/crashes like dereferencing an invalid pointer. I suspect from reading this post that that is the future Mr. Gregg has in mind.
> Whatever Cilium is, I cannot believe it generally prevents kernel crashes.
It doesn't magically prevent all kernel crashes from unrelated code. But what we can say is that Cilium itself can't crash the kernel unless there are bugs in the eBPF verifier.
My point is that the verifier could be relaxed to accept programs that never halt, thus not needing to solve the halting problem. You could then have the kernel just kill it after running over a certain maximum amount of time.
Why do you think the kernel crashes when crowdstrike attempts to reference some unavailable address (or whatever it does) instead of just denying that operation and continuing on? That would be the solution using this philosophy "just kill long running program". And no need for eBPF or anything complicated. But it doesn't work that way in practice.
This is just such a naive view. "We can prevent programs from crashing by just taking care to stop them when they do bad things". Well, sure, that's why you have a kernel and userland. But it turns out, some things need to run in the kernel. Or "just deny permission". Then it turns out some programs need to run as admin. And so on.
There is a generality in the halting problem, and saying "we'll just kill long runing programs" just misses the point entirely.
Likely what will happen is that you will kill useful long-running programs, then an exception mechanism will be invented so some programs will not be killed, because they need to run longer, then one of those programs will go into an infinite loop despite all your mechanisms preventing it. Just like the crowdstrike driver managed to bring down the OS despite all the work that is supposed to prevent the entire computer crashing if a single program tries something stupid.
> Why do you think the kernel crashes when crowdstrike attempts to reference some unavailable address (or whatever it does) instead of just denying that operation and continuing on?
Linux and windows are completely monolithic kernels; the crowdstrike agent isn't running in a sandbox and has complete unfettered access to the entire kernel address space. There is no separate "the kernel" to detect when the agent does something wrong; once a kernel module is loaded, IT IS the kernel.
Lots of people have indeed realized this is undesirable and that there should be a sandboxed way to run kernel code such that bugs in it can't cause arbitrarily bad undefined behavior. Thus they invented eBPF. That's precisely what eBPF is.
I don't know whether it's literally true that someday you will be able to write all possibly useful kernel-mode code in eBPF. But the spirit of the claim is true: there's a huge amount of useful software that could be written in eBPF today on Linux instead of as kernel modules, and this includes crowdstrike. Thus Windows supporting eBPF, and crowdstrike choosing to use it, would have solved this problem. That set of software will increase as the eBPF verifier is enhanced to accept a wider variety of programs.
Just like you can write pretty much any useful program in JavaScript today -- a sandboxed language.
You're also correct that due to the halting problem, we'll either have to accept that eBPF will never be Turing complete, OR accept that some eBPF programs will never halt and deal with the issues in other ways. Just like Chrome's JavaScript engine has to do. I don't really view this as a fundamentally unsolvable issue with the nature of eBPF.
The claim isn't that eBPF generally prevents kernel crashes. It's that it prevents crashes in the subset of programs it's designed for, in particular for instrumentation, which Crowdstrike is (in this author's conception) an instance of.
It's referring to Windows security software. If you have a lot of context with eBPF, which Gregg obviously does, the notion that eBPF will subsume the entire kernel doesn't even need to be said: you can't express arbitrary programs in eBPF. eBPF is safe because the verifier rejects the vast majority of valid programs.
It is not, programs that are accepted are proved to terminate. Large and more complex programs are accepted by BPF as of now, which might give the impression that it's now Turing complete, when it is definitely not the case.
I should clarify that individual eBPF programs have to terminate, but more complex problems can be solved with multiple eBPF programs, and can be "scheduled" indefinitely using BPF timers
I used to work for an EDR vendor and this post glosses over two major and important things.
1. There’s no need for eBPF on windows, it has the ETW framework (event tracing) which is much more powerful and provides applications subscribing to a class of events almost too detailed insights. the issue most AV vendors have with it though is speed. Leading to …
2. eBPF lets you watch. Congrats. It’s something, but it’s not the reason why these tools are deployed. Orgs deploy these tools to prevent or stop potentially bad stuff from executing. The only place this can be done in our operating systems is usually the kernel - for that you need kernel level drivers or various other filter drivers.
Crowdstrike screwed the pooch here, yes. But after a couple of days I feel like I haven’t read enough blog posts and articles that crap on Microsoft. It’s their job to build a secure operating system, instead they deliver Windows and because they themselves cannot secure windows, they ship defender… and we use tools like falcon like a bandaid for Microsofts bad security practices
> eBPF lets you watch. Congrats. It’s something, but it’s not the reason why these tools are deployed. Orgs deploy these tools to prevent or stop potentially bad stuff from executing
eBPF let's you prevent things too. seccomp filters can block syscalls.
The bigger problem is the performance you mentioned in 1. Crowdstrike's linux agent can work using eBPF instead of a kernel module, and will fall back to that if the current kernel version is more recent than the agent supports. But... then it uses up a lot more CPU.
This doesn’t seem grounded in reality. If you follow the link to the “hooks” that Windows eBPF makes available [1], it’s just for incoming packets and socket operations. IOW, MS is expecting you to use the Berkeley Packet Filter for packet filtering. Not for filtering I/O, or object creation/use, or any of the other million places a driver like Crowdstrike’s hooks into the NT kernel.
In addition, they need to be in the kernel in order to monitor all the other 3rd party garbage running in kernel-space. ELAM (early-launch anti-malware) loads anti-malware drivers first so they can monitor everything that other drivers do. I highly doubt this is available to eBPF.
If Microsoft intends eBPF to be used to replace kernel-space anti-malware drivers, they have a long, long way to go.
[1]: https://microsoft.github.io/ebpf-for-windows/ebpf__structs_8...