It's a matter of akrasia—if we can decide what makes us happy, then that implies that there's some sort of "meta-happiness"—a utility function that will be "best" maximized compared to other ones. Some people think that a utility function that doesn't include enjoying porn is better than one that does.

Of course, this is all theoretical if you can't change what makes you happy—and, since psychological studies say that how much you can change your utility function is something that varies individual-to-individual, there will never be a widely-accepted belief that it's impossible to decide what you like, as there will always be some people who can.

I'm not a crusader for or against porn (too cynical to crusade at all, really), but I think you've picked up the wrong end of the stick.

The problem in many people's minds is not with the consumers of porn but with the creators or actors of porn, especially the women. Like a lot of other so-called sex work fields, many human rights advocates end up torn: on the one hand, it's a field where women can make money with a certain amount of independence; on the other hand, the field can be violent, unsafe (drug addiction is rampant, sexually transmitted diseases are a constant fear, etc.) and (arguably) is enormously degrading to the spirit and the bodies of the women it employs.

Anyhow, I think that people who want to clamp down on porn are generally worried about the producers rather than the consumers. (In fairness, one influential argument from Catherine MacKinnon argues that porn makes the men who consume it more likely to be violent towards women. But I think that argument is the exception not the rule.)