I was mainly irritated that the ents had knees.

That the whole Sauron business was a local and temporary nuisance seems central to the roles of Tom B and the ents. The ents' complaint was, in the end, only with Saruman.