I suspect it's an embodied cognition thing. There is a very similar experience in musicians, where it's much much more difficult to reason about theory away from your instrument.

Even simple trivia type questions like "what's the V chord in F" are on basically instant recall for an experienced musician but without your instrument you have to stop and think (possibly even picturing your hands on your instrument) to answer.