Except we have excellent geological records on the matter so we know what weather was like mostly across periods. This isn’t really something we have been wrong on for a while. Ofcourse we can always be proven otherwise..
This is also backed by mass extinction events being relatively rare.