That is not accurate. Downvotes are for comments that negatively contribute to the conversation. One way a comment can negatively contribute is by failing to honor protocol, but it is not the only way. Even simply extraneous comments like lazy jokes probably should be and conventially are downvoted on Hacker News. Promoting pseudoscience definitely doesn't seem like a very good contribution to me.
> Maybe it is not a good contribution, but it is not for you alone to tell.
"Maybe it is an abusive, racist comment, but it is not for you alone to tell."
Your argument is hollow, because it applies to literally every reason to downvote. But technically you're right. Which is why we have voting. Collectively the readers decide which comments contribute to the conversation.
There is no moral obligation to give equal airtime to quackery and fraud.