From personal experience, it appears that this reaction occurs to all physicists, regardless of gender. Maybe men care less about this than women and are hence less easily disinterested in the field?