The speed of sound in air is ~330m/s so sitting about a metre away will produce ~3ms delay. However the speed of the vibration through human flesh (far denser than air) is about 4 to 5 times faster than the speed in air. You will "hear/feel" the sound transmitted through your body far faster than the sound transmitted through air. This maybe why I'm always seeing this magical "1ms" value.
The 10ms (or 5ms, depending on the source) is a threshold - under which a human perceives the sound as "instant", even though - yes, physically - it is not actually instant.
At 10ms (and possibly less), you will start to notice the lag and it will affect your performance.
https://www.soundonsound.com/sound-advice/q-how-much-latency...