I use to always click on newsletter unsubscribe links too. Then I wondered if they were taking a shot in the dark at a real user on the end of my email and me unsubscribing confirmed it.

Here's what I do now. If I don't remember ever subscribing to something, I look for the Unsubscribe button in the email client, which is part of the headers. It feels a little less "phishy" than the link in the email. It uses the message’s hidden “List-Unsubscribe” header, thus no tracking.