It's not even comfort for me, it's the hygiene of it. Touching greasy skin and then sharing... ick. You really need a personal device for everyone which is pricy.
I got these little fabric covers ("VR cover") for my HMD so the cleanliness aspect is a bit better managed. A couple people can use the first one and then I can swap it to a new, clean one, etc.
