>My use of "your" idea doesn't exclude your use of that idea
It can and does exclude me from certain uses of that idea. For example, if my intended use of that idea involves profiting from selling a good/service incorporating that idea, but my target customers decide not to buy from me specifically because you are using that idea. Sure, I can still use the idea in many other ways, same as you can use your land for many other things even whilst my sheep are grazing on it, and what was your wood for other things even after if I've burnt it to ash, but your use of my idea has reduced its value to me. All property is monopoly rights over something, and yes, all forms of property rights can be abused; that in itself is not an argument for their elimination.
The greater violation of liberty is preventing many entities from using an idea rather than a single entity losing the opportunity to profit from an idea.
Especially because a fire inventor being deprived of profit is not what is happening. What is happening is someone uses a broken system to register their idea of cooking rabbits after watching other people cook different animals. He is then given exclusive rights to cook rabbits, and he goes directly to extort shells and shiny rocks from the guy two caves down who has been cooking and trading rabbits for years now.
If we cannot have patents without "scan-to-email" extortionists we should not have patents. Creating an incentive to do no B2C business because it is more profitable to sue other companies for using obvious ideas is contrary to the intended purpose of the patent system anyway. Either get patent examiners that can do it properly, make it dirt cheap to challenge a patent, or shut it all down.
It can and does exclude me from certain uses of that idea. For example, if my intended use of that idea involves profiting from selling a good/service incorporating that idea, but my target customers decide not to buy from me specifically because you are using that idea. Sure, I can still use the idea in many other ways, same as you can use your land for many other things even whilst my sheep are grazing on it, and what was your wood for other things even after if I've burnt it to ash, but your use of my idea has reduced its value to me. All property is monopoly rights over something, and yes, all forms of property rights can be abused; that in itself is not an argument for their elimination.