That works pretty well if you're confident of finding another job and/or have a safety cushion (aka "FU" money).

In that situation, your manager doesn't really have much power over you. What's the worst they can do, fire you?

But a lot of people don't feel confident of finding another job and are basically afraid of losing the one they have.

That's what makes it a significant power difference. They're afraid because they aren't secure.