I mostly use emacs (having moved to it relatively recently) but fall back to vim when editing/viewing a small single file. On the odd occasion I do hit the wrong keys from misplaced muscle memory (and in vim a bunch of things could happen quickly) but usually I still have the Vim muscle memory and can manage. My main driver is emacs though.
I also use both and I found that keybindings and interaction with text are so different that it almost does not hurt. What I found much harder is using vim-like modes in editors - they almost match the vim keybindings but not just quite, luring you into the sense of familiarity but betraying at the most critical moment.
The only thing I've had to do was to map <C-x><C-s> in vim to ":w".
