Sure, "clean" as in doesn't do toilet/gross-out/sex humour, but a lot of his observational bits are "have you ever noticed how women are different from men in this way?"-type comparisons.
I can see how someone might find that kind of material sexist, especially if they've only encountered a few episodes where it seems like he's basically saying women are worse than men.
Personally, I don't have a problem with it because over lots and lots of episodes, whether he's punching down at men or women balances out. I'm not sure if the writers made a conscious choice to make it that way after maybe some initial criticism as I never followed the show when it was on air originally.
