Yes, we know eBPF must attach to equivalent events to Linux, but given there are already many event sources and consumers in Windows, the work is to make eBPF another consumer -- not to invent instrumentation frameworks from scratch.
Just to use an analogy: Imagine people do their banking on JavaScript websites with Google Chrome, but if they use Microsoft Edge it says "JavaScript isn't supported, please download and run this .EXE". I'm not sure we'd be asking "if" Microsoft would support JavaScript (or eBPF), but "when."
This assumes eBPF becomes the standard. It's not clear Microsoft wants that. They could create something else which integrates with dot net and push for that instead.
Also this problem of too much software running in the kernel in an unbounded manner has long existed. Why should Microsoft suddenly invest in solving it on Windows?
Microsoft has invested in solving this for at least two decades, probably longer. They are just using a different (arguably worse) approach to this than the Unix world.
In Windows 9x anti-malware would just run arbitrary code in the kernel that hooked whatever it wanted. In Windows XP a lot of these things got proper interfaces (like the file system filter drivers to facilitate scanning files before they are accessed, later replaced by minifilters), and the 64 bit edition of XP introduced PatchGuard [1] to prevent drivers from modifying Microsoft's kernel code. Additionally Microsoft is requiring ever more static and dynamic analysis to allow drivers to be signed (and thus easily deployed).
This is a very leaky security barrier. Instead of a hardware-enforced barrier like the kernel-userspace barrier it's an effort to get software running at the same protection level to behave. PatchGuard is a cat-and-mouse game Microsoft is always loosing, and the analysis mostly helps against memory bugs but can't catch everything. But MS has invested a lot of work over the years in attempts to make this path work. So expecting future actions isn't unreasonable.
This is a weird reading of history. Microsoft has spent tons of effort getting as much code out of the kernel as possible: Windows drivers used to be almost all kernel-mode, now they're nearly all in userspace and you almost never need to write a kernel-mode Windows driver unless you're doing something with deep OS hooks (like CS was, although apparently even that wasn't actually necessary). The safeguards on kernel code are for the tiny sliver of use cases left that need it, it is not Microsoft patching individual holes on the leaky ship.
They haven't yet gone as far as Apple in banning third-party kernel-mode code entirely, but I wouldn't be surprised if it's coming.
A thing I think a lot of people don't include in their premises about Crowdstrike is that they're probably the most significant aftermarket endpoint security product in the world (they are what Norton and McAfee were in 2000), which means they're more than large enough for malware to target their code directly, which creates interesting constraints for where their code can run.
I'm not saying I'd run it (I would not), just that I can see why they have a lot of kernel-resident code.
Microsoft made the reasonable point that locking 3rd parties out of the kernel might have resulted in legal challenges in the EU [0]. It is an interesting case where everyone is certain in hindsight that they would have been ok with MS blocking access, but it is less obvious that they would have taken that view if MS had pressured a bunch of security products out of the kernel with no obvious prompting.
The 2009 agreement with the EU mentioned in the article seems to be the one about the integration of the Internet Explorer (IE) into MS Windows.[1] But it only applied to IE and the commitment was limited to 5 years.[2]
Or is the article referring to something else?
I see no reason why the EU should object to Microsoft's adoption of eBPF as long as MS Defender simply uses the same API that is available to all competitors.
Apple took the lead on this front. It has closed easy access to the kernel by apps, and made a list of APIs to try and replace the lost functionality. Anyone maintaining a kernel module on macOS is stuck in the past.
Of course, the target area of macOS is much smaller than Windows, but it is absolutely possible to kick all code, malware and parasitic security services alike, from accessing the kernel.
The safest kernel is the one that cannot be touched at runtime.
I don't think Microsoft has a choice with regards to kernel access. Hell, individuals currently use undocumented NT APIs. I can't imagine what happens to backwards compat if kernel access is closed.
Apple's closed ecosystem is entirely different. They'll change architectures on a whim and users will go with the flow (myself included).
But Apple doesn’t have the industrial and commercial uses that Linux and Windows have. Where you can’t suddenly switch out to a new architecture without massive amounts of validation costs.
At my previous job they used to use Macs to control scientific instrumentation that needed a data acquisition card. Eventually most of the newer product lines moved over to Windows but one that was used in a validated FDA regulated environment stayed on the Mac. Over time supporting that got harder and harder: they managed through the PowerPC to Intel transition but eventually the Macs with PCIe slots went away. I think they looked at putting the PCIe card in a Thunderbolt enclosure. But the bigger problem is guaranteeing supply of a specific computer for a reasonable amount of time. Very difficult to do these days with Macs.
Good luck getting that through a regulated company’s Quality Management System or their legal department. Way too much business risk and the last thing you want is a yellow or red flag to an inspector who can stop ship on your product until all the recall and remediation is done.
See Satya Nadella has recently said that Microsoft will now put security above any other value at Microsoft. He specifically even singled out backwards compatibility.
Microsoft is a big boat. It takes a long time to turn if the captain orders it. But the captain has ordered it. I expect the sacrosanct nature of back compat to eventually die. Windows will never turn into the moving target that macOS is, but I expect a lot of old stuff to just stop working, when security concerns dictate they should be removed.
> The safest kernel is the one that cannot be touched at runtime.
Can you expand what you mean here? Because depending on the application you are running, you will need at least talk with some APIs to get privileged access?
Yeah, Apple doesn’t allow any user code to run in kernel mode without significant hoops (the kernel is code signed) and tries to provide a user space API (e.g. DriverKit) as an alternative for the missing functionality.
...just like they did with Kerberos! And just like with Kerberos they'll define a standard then refuse to follow it. Instead, they will implement subtle changes to the Windows implementation that make solutions that use Windows eBPF incompatible with anything else, making it much more difficult to write software that works with all platforms eBPF (or even just its output).
Everything's gotta be different in Windows land. Otherwise, migrating off of Windows land would be too easy!
In case you were wondering what Microsoft refused to implement with its Kerberos implementation it's the DNS records. Instead of following the standard (they wrote!) they decided that all Windows clients will use AD's Global Catalog to figure out which KDC to talk to (e.g. which one is "local" or closest to the client). Since nothing but Windows uses the Global Catalog they effectively locked out other platforms from being able to integrate with Windows Kerberos implementation as effectively (it'll still work, just extremely inefficiently as the clients won't know which KDC is local so you either have to hard-code them into the krb5.conf on every single device/server/endpoint and hope for the best or DNS-and-pray you don't get a Domain Controller/KDC that's on an ISDN line in some other country).
This to me ascribes way too much to mustache-twirling villainy at Microsoft, but to me fails to account for the fact that engineers surely make many of these implementation-detail decisions. These engineers aren’t incentivized to create lock-in. I think it’s more likely that sometimes for a feature to play well with other existing parts of the Windows ecosystem, compromises are made to the standards-compliance. Microsoft may have shipped those related interfaces before this standard had been hashed out, so they have a choice to break everything or to not perfectly follow the standard.
Note: I’m not a Windows dev so I can’t speak to specifics of anything like your Kerberos example. I just don’t believe MS is full of evil engineers, nor that Satya Nadella visits cubicles to promote lock-in practices.
> These engineers aren’t incentivized to create lock-in.
Ever heard of something called “money”?
> I think it’s more likely that sometimes for a feature to play well with other existing parts of the Windows ecosystem, compromises are made to the standards-compliance.
So you're basically saying that you're too young to remember the “good” old days of Embrace, Extend, Extinguish, right...?
This doesn't really seem like their strategy anymore. It's not like Edge directly interprets Typescript, for example. While they embraced and extended Javascript, any extinguishing seems to be on the technical merits rather than corporate will.
In the case of security scanners that run in the kernel, we learned this weekend that a market need exists. The mainstream media blamed Crowdstrike's bugs on "Windows". Microsoft would likely like to wash its hands of future events of this class. Linux-like eBPF is a path forward for them that allows people to run the software they want (work-slowers like Crowdstrike) while isolating their reputation from this software.
> Why should Microsoft suddenly invest in solving it on Windows?
If they can continue to avoid commercial repercussions for failing to provide a stable and secure system, then society should begin to hold them to account and force them to.
I’m not necessarily advocating for eBPF here, either. If they want to get there through some “proprietary” means, so be it. Apple is doing much the same on their end by locking down kexts and providing APIs for user mode system extensions instead. If MS wants to do this with some kind of .net-based solution (or some other fever dream out of MSR) then cool. The only caveat would seem to be that they are under a number of “consent decree” type agreements that would require that their own extensions be implemented on a level playing field.
So what. Windows Defender shouldn’t be in the kernel any more than CrowdStrike. Add an API. If that means being able to send eBPF type “programs” into kernel space, cool. If that means some user mode APIs, cool.
Not necessarily disagreeing with you, but as far as 'avoiding commercial repercussions' goes... Windows' share of the desktop OS has market has been declining for almost 20 years at the rate of about 1% per year. And about 70% of the global installed base is still on Windows 10.
They have a long way to fall, but I'm not sure that if I'm a regulator I look at that and say there needs to be some kind of intervention by society apart from what market forces are gradually doing anyway.
Windows development on eBPF is slower than Linux development on eBPF, so it will never be supported. A source code user licensee could develop it faster, but who licenses Windows source and already has great eBPF experience?
Right, and we wanted to talk about all security solutions and not make this about one company. We also wanted to avoid shaming since they have been seriously working on eBPF adoption, so in that regard they are at the forefront of doing the right thing.
Right; disabling eBPF doesn't solve this. And the bigger point is that this kind of eBPF is still super-user only.
Apart from the more exotic facilities, the critical facilities that would be hard to disable include LD_PRELOAD for interposers/shims (as you mentioned), and gdb for just setting breakpoints on crypto functions. And if neither of those existed, then I may have to edit openssl code and recompile my own edited version. And if that wasn't allowed (signed libraries) then maybe I'd edit the application code or binaries.
Why don't recommend atop? When a system is unresponsive, I want a I want a high-level tool that immediately shows which subsystem is under heavy load. It should show CPU, Memory, Disk, and Network usage. The other tools you listed are great, once you know what the cause is.
My preference is tools that give a rolling output as it let you capture the time-based pattern and share it with others, including in JIRA tickets and SRE chatrooms, whereas top's generally clear the screen. atop by default also sets up logging and runs a couple of daemons in systemd, so it's more than just a handy tool when needed, it's now adding itself to the operating table. (I think I did at least one blog post about performance monitoring agents causing performance issues.) Just something to consider.
I've recommended atop in the past for catching short-lived processes because it uses process accounting, although the newer bpf tools provide more detail.
For a busy 64-CPU production JVM, I tested Google's Java symbol logging agent that just logged timestamp, symbol, address, size. The c2 compiler was so busy, constantly, that the overhead of this was too high to be practical (beyond startup analysis). And all this was generating was a timestamp log to do symbol lookup. For DWARF to walk stacks there's a lot more steps, so while I could see it work for light workloads I doubt it's practical for the heavy production workloads I typically analyze. What do you think? Have you tested on a large production server where c2 is a measurable portion of CPU constantly, the code cache is >1Gbyte and under heavy load?
I regularly profile heavy time-sensitive (as in: if the code takes too long to run it breaks) workloads, and I even do non-sampling memory profiling (meaning: on every memory allocation and deallocation I grab a full backtrace, which is orders of magnitude more data than normal sampling profiling) and it works just fine with minimal slowdown even though I get the unwinding info from vanilla DWARF.
Granted, this is using optimized tooling which uses a bunch of tricks to side-step the problem of DWARF being slow, I only profile native code (and some VMs which do ahead-of-time codegen) and I've never worked with JVM, but in principle I don't see why it wouldn't be practical on JVM too, although it certainly would be harder and might require better tooling (which might not exist currently). If you have the luxury of enabling frame pointers then that certainly would be easier and simpler.
(Somewhat related, but I really wish we would standardize on something better than DWARF for unwinding tables and basic debug info. Having done a lot of work with DWARF and its complexity I wouldn't wish it upon my worst enemy.)
From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over"
While it might call itself a benchmark, it behaves very microbenchmark-y.
The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation.
The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by:
Significant disk input/output
Moderate system and application execution time
Transaction integrity
TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A.
...
Transactions are submitted by programs all executing concurrently.
I don't think I have. I was only responding to the factually incorrect statement of yours that pgbench is a microbenchmark.
> which was about whether databases could even have micro-benchmarks.
No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there.
I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not.
> You also missed the word "Obsolete"
I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
I didn't pull this argument out of nowhere, please read the direct comment I was replying to. Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs."
I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?
> I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?
Wow, just wow. How ridiculous this is?
> Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs."
My only position in the whole thread was "1% of overhead cannot be universally claimed" and I still stand by it 100%. pgbench experiment from Linux kernel folks was just one of the counter-examples that can be found in the wild that goes against your claim. And which you first disputed by saying that it is a micro-benchmark (meaning that you have no idea what it is) and now you're disputing it by saying it's obsolete (yes, but still very relevant in database development, used in the Linux kernel and not the slightest technical reasoning after all).
Personally, I couldn't care less about this but if names is what you're after, you're not disagreeing with me but with the methodology and results presented by Mel Gorman, Linux kernel developer.
You are backing away from your other positions, for example:
> I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
Do you now agree that TPC-B is too simple and microbenchmarky? And if not, please tell me (as I'm working on the problem of industry benchmarking in general) what would it take to convince someone like you to stop elevating obsoleted benchmarks like TPC-B? Is there anything?
A major postgres contributor (Andres Freund) disagreed with you about pgbench but, yes, feel free to dismiss them just because you found some words on a web page.
I am just a minor PostgreSQL contributor and consultant of no import, but do you seriously think you are superior to PostgreSQL core devs and know more than them about PostgreSQL just because you are a Linux kernel dev? I really do not like your attitude here and your appeal to your own authority when you are not even an authority here.
Pgbench is used heavily both in the development of PostgreSQL and by people who tune PostgreSQL. So it is not obsolete. Maybe it is a bad benchmark and that the PostgreSQL community should stop using it but then you need a stronger argument than just some claim about obsoleteness from 1995. If a PostgreSQL core developer says in 2024 that it is still relevant I think that weighs a bit higher than a claim from 1995.
Yes, indeed, it is very ridiculous to pull out arguments such as "who are you". I mean, wtf?
Your replies demonstrate lack of technical depth in certain areas and which makes me personally doubt in your experiment results. And you know what, that is totally fine.
But your attitude? A total disbelief.
> Do you now agree that TPC-B is too simple and microbenchmarky?
No, why would I, you did not present any evidence that would support that claim of yours?
And contrary to you, and to your own embarrassment, I do not use personal attacks when I go out of technical depth.
> what would it take to convince someone like you to stop elevating obsoleted benchmarks like TPC-B? Is there anything?
You don't have to convince me anything. Remember that this is just a random internet page where people are exchanging their opinions.
Initially we just turned off frame pointers for the Python 3.9 interpreter in Fedora. They are back on in Python 3.12 where it seems the upstream bug has been fixed, although I can't find the actual fix right now.
I don't know where 1-2% comes from, but for many scale production workloads I studied it was so close to 0% that it was tough to measure beyond noise on the cloud. That's not to say that 1-2% is wrong, but that it's likely someone's workload and other people see less.
Helping people find ~30-3000% perf wins, helping debugging and automated bug reports, is huge. For some sites it may be like 300 steps forward, one step back. But it's also not the end of the road here. Might we go back to frame pointer ommision one day by default if some other emerging stack walkers work well in the future for all use cases? It's a lot of ifs and many years away, and assumes a lot of engineering work continues to be invested for recoving a gain that's usually less than 1%, but anythings possible.
There's a couple of problems with an apt reinstall. One is that people often don't work on performance until the system is melting down -- many times I've been handed an issue where apt is dreadfully slow due to the system's performance issue and just installing a single package can take several minutes -- imagine reinstalling everything, it could turn the outage into over an hour! The other is I'd worry that reinstalling everything introduces so many changes (updating library versions) that the problem could change and you'd have no idea which package update changed it. If there was such an apt reinstall command, I know of large sites (with experience with frame pointer overheads) that would run it and then build their BaseAMI so that it was the default. Which is what Ubuntu is doing anyway.
That's just the same handwavey reason as given in the article. Where is the evidence that this will actually result in any significant amounts of optimization that wouldn't be possible without making everything (slightly) slower for end users.
I've had a few CO2 meters, and learned not to trust any <$100 as they don't really work. I look for meters that use non-dispersive infrared diffusion sensors (NDIRs), like the Aranet4. I've found the TIM10 desktop model from co2meter.com to be accurate (AFAICT), uses NDIRs, and only US$139.
I also have other air quality meters. (I collect measuring devices.) I wish there was a do-it-all air meter.
I have a TIM10 that was once reporting negative CO2 levels one spring with the windows open. I'm not sure what I think about it. It's not a scam product by any means, it can definitely tell when someone is in the room or a window is opened or closed. I needed the space and unplugged it awhile ago and just don't worry about CO2 levels currently.
My understanding is that some co2 sensors assume the minimum value seen over the last 30 days is 400ppm as a calibration. This could possibly explain a negative value if the sensor has been in higher co2 environment for some time, and is suddenly exposed to the outside.
Do you know if the Aranet4 can be used over bluetooth/wifi without internet access? I've been looking at a bunch of devices and haven't seen aranet yet.
Also, have you ever put all of your devices near each other to compare the variance between them all?
I got my Aranet4 because it had very good reviews and allowed local read out of sensor data without any "cloud" involvement. I've been very happy with it, the readings are integrated into my Home Assistant setup so automations can respond to high CO2 (e.g. turn on a fan directed at a window, when temperatures are high enough to have a window open).
Just to use an analogy: Imagine people do their banking on JavaScript websites with Google Chrome, but if they use Microsoft Edge it says "JavaScript isn't supported, please download and run this .EXE". I'm not sure we'd be asking "if" Microsoft would support JavaScript (or eBPF), but "when."