The name of the Washington football team was also used "without any thought", and that there was no need to change it as it wasn't meant as a slight against the group for whom the name is a slur. But it turns out that these things can be hurtful even if you don't mean them in a nasty way.

But yeah, "off in left field" is a baseball thing; no one is harmed or offended by that— even if you do apply it to politics or use it in a political context, as the WP article briefly mentions, politics are something you choose, not your cultural identity, so it really isn't the same thing at all: https://en.wikipedia.org/wiki/Out_of_left_field

Another interesting case is the phrase "balls to the wall" which has a harmless origin with aeronautics, but is spicy enough on account of an obvious alternative interpretation that it isn't used in polite company.

The first is a phrase steeped in the genocide of indigenous people.

Off in left field is a baseball analogy and has nothing to do with people left of center politically.