The vast majority of that first graph happened more than 1000 to 10000 years ago. Attention to endangered species seems to have only started 50 to 100 years ago.

But you are right that our impact on endangered species is poor; only 28 species have recovered out of ~2000 listed in the US (according to the Wikipedia page for the Endangered Species Act of 1973).