I do this rather frequently by mistake. I, like you, try to point out my error so the OP doesn't wonder why they are being downvoted.
Another problem is that I often upvote articles by scrolling on my phone. In addition to providing false signal, this has the unfortunate side effect of polluting my "saved stories".
I wish there was a mechanic to change votes after submission (even if the ability to change your vote was only momentary).
