That's an entirely reasonable skeptical point of view to take in the absence of other evidence. However, this question has also been examined. Using typical measures of socioeconomic status (e.g. an index of education + income), one finds that controlling for socioeconomic status does little to explain the black-white IQ gap. There is a sizable gap at every level of socioeconomic status. Data: http://sites.biology.duke.edu/rausher/Hm2.jpg
It's still ridiculous to tie it to 'race' because, a) there is no real biological classification of race, which means all of this data using such classifications is already inherently biased, and b) mere correlation again does not entail a causation, so there could very well be a third variable that ties it all together. I can think of a bunch right off the top of my head, like stereotype threat and the still all too common racism resulting in many educational disadvantages for black students.
source: http://sites.biology.duke.edu/rausher/lec23_05.html