Once you start to get a sex imbalance then the genetics of future generations as a whole will disproportionately be determined by the female population, which won't have any copies of the gene. Eventually the gene will be eliminated from the population. It's the same principle that causes most species to produce equal number of males and females despite that not being ideal for the species as a whole.