You're greatly underestimating what people can manage on their own. Even ancient peoples managed plenty of harm to nature. Deforested whole regions, wiped out species, damaged farmland with irrigation, etc.
Now there are vastly more of us, and we have a lot better tools to do damage with.