Again like you said, the reason not to wear is nothing specific to watches. At the high end of collectibles markets, condition grading is extremely fine and detailed. A knowledgeable person can see the difference between something that has never left the box and something that was used/worn once. Additionally, there's often a substantial value difference between a #1 condition item and a #2 condition item. As the participants in any collector market become more sophisticated, condition difference between individual items becomes both more legible (as grading methods are created) and has a greater effect on value (as the market expands).

There's also a sort of status game being described in the part you quoted. Self-identified collectors/dealers intentionally don't wear their collection in order to signal their understanding of the watches' value and therefore their own knowledge as connoisseurs.