For quick browsing of a files commits I use Emacs with Magit and utilize the time travel feature of Git-Timemachine https://github.com/pidu/git-timemachine
You don't get the annotations you want, but you could make a comment with a common starting word like NOTE: then remove them before you merge the refactorings.