It's helpful socially to be able to check the time subtly. A tap or large wrist motion makes it look like you're trying to tell whoever you're with that they should be paying attention to the time.
I understand what you're saying. However, I don't see how that's much different from glancing at any number of clocks: for instance at the office when I glance at the time in the corner of my computer screen, when I'm in a meeting room and glance at the clock on the wall, etc. People notice that motion all the time.
I notice when people do this to me as well. Either way you'd need to glance at your watch, which means your eyes move somewhere else and you lose eye contact with whoever you're with/speaking to.
