Many times, a comment is so inane that the only time-efficient way to express disagreement is to vote against it.
Any active forum is going to have a tiny bit of brass buried in a large pile of muck. If we spend all of our time apologizing to the muck-makers, there's very little time to find the brass.
If someone says something I don't agree with, but says it reasonably and with some sort of support, I'll reply instead of downvoting.