Vimperator is the king among Vim-like keyboard shortcut enabler extensions in browsers, sadly it may not work in the future versions of Firefox because Firefox is transitioning from highly free but less safe/performing XUL based extensions to a safer but limited WebExtension model. It is very fast and smooth with lots of features.
I used vimperator and experienced crazy slowdown on Firefox in both OSX and Ubuntu. This happens after you browse for a while. Wasn't sure why initially. Things improved dramatically after reinstalling the addon. But I had to do it frequently so it wasn't worth it.
I think it's because things are saved in history for a long time or history chain of sites in a tab is very long.
Edit:
Only extension that is vim-like that hasn't let me down is cvim for chromium/chrome.
It's true that some sites are harder to navigate, though.