Part of the issue is that it's much easier to ask somebody about their age, gender, race, sexual orientation, etc (at least in the context of a survey or something) than it is to ask them "Are you ugly? Y/N". It's not just a category, it's a value judgment. This also drastically reduces the number of ugly advocacy groups and advocates. It's hard to push for better treatment of a group in which almost nobody wants to claim membership.