Yes, the fault lies with intolerant minds.

If we forget this, then intolerant minds will have awesome veto powers over what the rest of us will be allowed to discuss.

Intolerant people are likely to use those veto powers.