"Free speech" does not mean "free from all consequence," and never has.

What do yo really mean by that?

Personally I worry that mere attaching some empty labels to an argument is reason enough justify censoring it.

I mean that if you choose to use slurs or donate money to hate groups, and are ostracized from a community as a result, then invoking "free speech" is not sufficient defense to be allowed back into that community.

Or, in popular comic form: https://xkcd.com/1357/

So "you are free to say whatever you want, but if I don't like it I will destroy your life".

I'm not sure whose life was destroyed or how we jumped to that point, but yes, if you choose that slurs and hate groups are more important to you than your community's acceptance, then you get to deal with those consequences. At no point is this an infringement on your right to free speech.

Well, that's horrifying. I'll shut up now.