Good point. Though a significant part of typing speed is your _spacial_ memory (where is the J key on the keyboard?).
Your _muscle_ memory makes you good at moving your fingers (or thumbs) to an absolute position on your keyboard (or screen) - but to do that you must first decide what absolute position your finger (or thumb) should move to (where is the J key?).