Sex and race are just treated differently by law. The supreme court wouldn't allow drafting only black people, but it does allow drafting only men.

The rules about sex-based discrimination were not crafted for the purpose of college admissions. This is just how the law has evolved around treating people of different sexes and races differently.

Another consequence of having the looser "intermediate scrutiny" for sex discrimination is that the FBI can have different fitness requirement for male and female recruits. Ditto for armed forces, and for laws relating to pregnancy.