Perhaps, but there are other options -- international bus tickets are pretty cheap. So is the ferry between, for example, Morocco and Spain. Plenty of people could afford this, or have friends/relatives in wealthy countries who could pay for them.

It is actually easier to visit some countries by air today. Visa on arrival, visa-free transit or stay—these options can be available only at airports.

Passport control as immigration regulation/deterrent definitely rings true to me, and some countries see less need for it with richer travelers.