Usually when people say 'swapping' they mean page faulting. It's nothing more than a slight annoyance on a single-user machine if you swap for 10 seconds, but on a busy server you are dead in the water.
Take this with a grain of salt since this anectode is a few years old, before I upgraded to a SSD to host my swap, and this is on Windows.
I'll always remember when I used to load a large piano instrument in a VST DAW on Windows 7, taking about 3-4GB out of 12GB of RAM, it played perfectly fine but if I left the application open, invariably on the next day I'd get a barrage of audio dropouts when pressing any new piano key. One trick was to put my arms on the entire keyboard a few times to wake the swap back up to memory. Another trick, which I ended up relying upon despite occasional low memory warnings, was to disable the pagefile entirely - that sure fixed the problem.
I'm not sure how/if things have improved since with Windows 10 and SSDs, but I always felt there was something wrong with the algorithms, since even with GBs of memory free at all times, old memory content would tend to end up on disk, without any good reason I could see.
I assume the OS used time to prioritize various caching/pre-fetching techniques over actual application data, and/or once paged, never preemptively loaded data back to RAM even if plenty of memory was available.
