I would imagine that the amount of available calories is overall the larger influence on which society grows and which doesn't. Remember that in most hunter-gatherer and farming societies, population stayed constant despite high birth numbers.
This is kind of mentioned in the interview I posted in the sibling reply, but I could imagine that this mainly drives down the density, but not so much the rate of growth. The reason is that grown individuals can actually usually feed themselves by roaming. What is hard is achieving a caloric surplus, which is why there is little specialisation and social stratification.
