Reminds me of sites that required ActiveX to run arbitrary code on the user side when visiting a web site outside a sandbox. Turned out to not be ideal from a security point of view.
But I guess `ssh -X` users still miss those times...
I suspect you don't really guess that. There are differences between the two cases. For example, security threat models are a thing. Something can be secure against the threats it will face. I don't ssh -X into servers I don't already trust. There is no arbitrary code being run.