Exactly. Returning from orbit is "easy", you just go into a free-return trajectory and don't even do much of a burn at Mars, if any at all.
Getting a sample from the surface basically means you need to do a deceleration burn to get into orbit, and then the reverse after grabbing the sample (which itself must be accelerated to orbital speed). No small thing.
A free-return trajectory doesn't enter orbit around a body, indeed its appeal is that if your propulsion fails while on such a course, you'll coast home "for free" rather than enter orbit around your target or drift into deep space.
Entering orbit almost always requires an orbital insertion burn, and in the case of an Earth-Mars Hohmann transfer definitely does.
Your point stands though: returning from a fly-by (including a free-return) requires much less fuel than returning from orbit, and even that's much easier than returning from the surface.
For fun, take a look at rotovators (AKA "momentum exchange tethers") that could make orbital capture/departure—and even landing/launch—"free" on bodies with minimal atmosphere.
Getting a sample from the surface basically means you need to do a deceleration burn to get into orbit, and then the reverse after grabbing the sample (which itself must be accelerated to orbital speed). No small thing.