Indeed. Chinese isn't really more difficult so much as it's different. And the difference causes people to think it's difficult, which then becomes an excuse for poor results when teaching/studying the language. Though I will say it tends to be poorly taught, so I guess it's more difficult in that sense (people will struggle with anything that's poorly taught).
And it's poorly taught because there is a lack of people who really master the two languages (Chinese + language X that is very different from Chinese), which is due to the fact that... ...the two languages are very different in the first place...
I don't think someone would need to be a master of both languages to greatly improve the way it's taught. For instance, one of the problems is that characters tend to be taught to foreigners in a completely bizarre manner; you end up learning many compound characters before you learn the component parts. Other issues include the small amount of Chinese in the course (lots of speaking in English about Chinese), the focus on things like grammar, and the idea that Chinese is so difficult that poor results are acceptable.
I've seen a number of people with a low-level in Chinese develop self-study plans that were much, much more effective than class or text book I've seen.
