Yes. After initial infection, which usually presents asymptomatically or with flu-style malaise, the parasites form cysts in the brain and stay there, where they aren't transmissible. The danger to the fetus comes with a new infection in the transmissible phase, which is likely to disrupt the fetal nervous system in lethal or severely damaging ways to which an adult brain isn't so susceptible.

(Source: I read a lot. Could be wrong! Don't think I am.)