As long as the other sex also has the same opportunity to join a similar group, I agree.

But when there is no proper alternative for a man or woman to join a similar club, it's more about the exclusion of people with the same interest but happen to have the wrong sex.