To illustrate why even reasoning from general principles has issues, consider: are you also against punishing people who commit crimes, on the grounds that almost nobody wants to be punished?