I have similar experiences. I've been digging into this more over the years and my two conclusions are: (a) Linux memory management is overall rather complex and contains many rather subtle decisions that speed up systems. (b) Most recommendations you find about it are old, rubbish, or not nuanced enough.
Like one thing I learned some time ago: swap-out in itself is not a bad thing. swap-out on it's own means the kernel is pushing memory pages it currently doesn't need to disk. It does this to prepare for a low-memory situation so if push comes to shove and it has to move pages to disk, some pages are already written to disk. And if the page is dirtied later on before needing to swap it back in, alright, we wasted some iops. Oh no. This occurs quite a bit for example for long-running processes with rarely used code paths, or with processes that do something once a day or so.
swap-in on the other hand is nasty for the latency of processes. Which, again, may or may not be something to care about. If a once-a-day monitoring script starts a few milliseconds slower because data has to be swapped in... so what?
It just becomes an issue if the system starts trashing and rapidly cycling pages in and out of swap. But in such a situation, the system would start randomly killing services without swap, which is also not entirely conductive to a properly working system. Especially because it'll start killing stuff using a lot of memory... which, on a server, tends to be the thing you want running.
Default configs of most distros are set up for server-style work, even on workstation distros. So they’ll have CPU and IO schedulers optimized for throughput instead of latency, meaning a laggy desktop under load. The whole virtual memory system still runs things like it is on spinning rust (multiple page files in cache, low swappiness, etc).
The only distro without this problem is Asahi. It’s bespoke for MacBooks, so it’s been optimized all the way down to the internal speakers(!).
> Default configs of most distros are set up for server-style work, even on workstation distros. So they’ll have CPU and IO schedulers optimized for throughput instead of latency, meaning a laggy desktop under load. The whole virtual memory system still runs things like it is on spinning rust (multiple page files in cache, low swappiness, etc).
LOL. A Ken Colivas problem, circa 2008, still there :-)))
Like one thing I learned some time ago: swap-out in itself is not a bad thing. swap-out on it's own means the kernel is pushing memory pages it currently doesn't need to disk. It does this to prepare for a low-memory situation so if push comes to shove and it has to move pages to disk, some pages are already written to disk. And if the page is dirtied later on before needing to swap it back in, alright, we wasted some iops. Oh no. This occurs quite a bit for example for long-running processes with rarely used code paths, or with processes that do something once a day or so.
swap-in on the other hand is nasty for the latency of processes. Which, again, may or may not be something to care about. If a once-a-day monitoring script starts a few milliseconds slower because data has to be swapped in... so what?
It just becomes an issue if the system starts trashing and rapidly cycling pages in and out of swap. But in such a situation, the system would start randomly killing services without swap, which is also not entirely conductive to a properly working system. Especially because it'll start killing stuff using a lot of memory... which, on a server, tends to be the thing you want running.