The crucial difference is "the right to swing your arms ends at my nose" essentially. The actual impact on other people is what makes the limitations acceptable.

That is an important distinction compared to random searches because you "might" have done something wrong.