IMHO when people say 'math is hard' they mean 'it takes more work than other subjects to be good at' - you're either a prodigy or you grind problems until you get the intuition. The easier subjects you can usually talk everyone and yourself into thinking you know them, or perhaps the ratio of memoization to practice is skewed more towards memoization. Maths is practice, practice, practice and then some more practice - blood, sweat and tears.
I’m not even sure it takes more work than, say, getting good with language. Hell, it might take less!
I think the main difference is that practicing language is far more rewarding for most people, than practicing math. They also have way more opportunities to practice it naturally, without even intending to do so.