If it was true that agriculture has caused a substantial slide in intelligence, remnant hunter-gatherer populations, and populations that were hunter-gatherers until recently, should be the smartest people on earth. For example, Native Americans (at least the ones from tribes which weren't too big on agriculture) should be smarter than most other ethnic groups in America. As far as I know, that's not the case.