For a desktop with sufficient amount of memory swap is just bad from my experiences. When i was using swap on my desktop system, which i usually have running 24/7, i'd often come back in the morning to find that linux decided to swap out my X session or something stupid like that. How much of my 16GB memory will a "rarely used daemon" use? Insignificant compared to how annoying it is to have to wait for your interactive applications to be swapped in when that happens.
It sounds like you had a process running over night that needed enough memory to force X (or whatever) to swap. Without the swap, that memory-hungry process would presumably have run less optimally (if it got memory allocation failures and adjusted its behavior to utilize less memory) or it would have simply died. It's not so clear that having processes die because they can't get memory is better than having something else run slowly, in general.
Edit: Possible caveat: Maybe whatever was running overnight did a lot of disk I/O and the OS decided to cache it at the expense of moving idle processes to swap (not sure if Linux does that or not).
"Maybe whatever was running overnight did a lot of disk I/O and the OS decided to cache it at the expense of moving idle processes to swap (not sure if Linux does that or not)."
In my experience (Ubuntu 9.10, kernal 2.6.31), with the default settings it does not do this during intensive disk I/O unless there is very little free memory to start with.
In other words, if the system is not low on memory to start with, I can start a bunch of processes that read and/or write multiple terabytes of data and when I come back the next morning little or no additional swap will have been consumed.
