Height does make phones less comfortable to handle with a single hand, which is what I prefer to do most of the time. They are heavier, lean further back so you have to spread your fingers on the back more, and reaching the top to pull the notifs bar, for example, is harder.
I'd argue height is worse than width in comfort.