It probably depends on what you are comprising about, and how much scope for a freely negotiated deal there is in that subject matter. In general, negotiation rather than compulsion makes both parties to the negotiation winners.

I'm pretty much a strong advocate of freedom of speech, having lived in countries without it, but just for mental exercise I'll try disagreeing with the example you give, which I agree is well chosen to force agreement with your much more general point.

What if Group A says, "We should have a right to free speech to impugn your mother's moral character," and Group B says, "We like free speech in general, but we think defamation should still bear legal penalties." Is your example as strong if it is that concrete?

The difficulty caused by introducing 'principles' like justice. Free speech is such a principle.

Principals are compromised by compromise. Preferences are not. Therefore if you think in terms or the right to freedom of speech, compromise is how you lose. If dislike restrictions on your freedom to say what you want, negotiation is how you win.