Sex encompasses more than the production of some gamete; it includes genetic and anatomical determinants, and it ain't exactly unheard of for those to get out of sync or otherwise produce sexes that are neither unambiguously male nor unambiguously female.

This is also why sex and gender are modernly regarded as distinct from one another, on that note.