More than being black, which I'm sure it contributes, I think he was a bit of an outsider in academia. Typically you would make friends and connections so your work gets more spread out. It seems that sowell was a bit on his own.

There's a lot of nepotism in academia, only the filter of time seems counterbalance it.