My impression is that IQ seems to be a measurement of a person's capacity to think abstractly, which is something you learn through the world around you. Perhaps those who fail at math have simply never experienced anything that requires abstract thought. Therefore, it seems like there is a skill that even basic arithmetic is dependent upon, basic abstractions. I can see how an otherwise able person would be hopeless at math if they never learned abstract thought.
"Perhaps those who fail at math have simply never experienced anything that requires abstract thought."
I disagree. Most math classes are structured as far away from the abstract as possible. Math classes, at least in the US high schools, are simply students sitting down and crunching numbers on paper with 40 other people around you. How is that abstract thought? I'm not saying math isn't useful in the abstract, I'm saying the way we measure people's math abilities is too focused.
Crunching numbers is most definitely abstract. What is a number? Why does it matter? What does it represent? What is zero? Why is zero important? Think about how long it took for humans to actually start doing math. And then think about how difficult it used to be for people to grasp negative numbers. Sure, schools teach things with M&Ms and counting other concrete things, but you still have to think abstractly to apply the same concepts to other concrete things.