I don't buy that. That's like saying some people are smart enough to read, but just can't figure out writing.
There will always be those with exceptional aptitude. But if you must be exceptional to simply understand a subject as broad as computer science, that's a problem with how we conceive the science.
The Romans thought multiplication was really hard, too. Exponents were literally unthinkable. We know better not because we're that much smarter, but because we have a simpler and more powerful way to think about it.
This is perhaps the biggest obstacle for people who are trying to teach others to program - and so I agree with you here.
However, it does seem like there are some people who will just never be able to pick it up - even though they are demonstrably smart in other areas.