Everyone is the direct descendant of Africans. That's not what "Black" means in the sense it's being used. The people you are talking about are culturally Dravidian Indian, not culturally African-American.

"black" as commonly used in America refers to people of sub-Saharan descent. It does not have much to do with actual skin color.