I had the same experience. German was my second, Chinese Mandarin my third. The first two years I was learning Mandarin, I found reading, writing and listening were fine. However, when it came to actually speaking, I was constantly stuttering because my brain wanted to substitute in German words instead of Mandarin words.

My girlfriend at the time natively spoke two languages (English and Hokkien) but was less proficient in both than many people who were only native speakers of one of them. She did, however, manage to pick up Mandarin a whole lot easier than I did.