At the same time, I remember most of high school math barely needing calculators outside chemistry and physics.

Look at some of the SAT math questions:

https://satsuite.collegeboard.org/media/pdf/digital-sat-samp...

The questions are all designed to have a tidy, closed-form answer. A calculator is either marginally helpful or outright cheating.