I tend to vary the amount of refactoring I do depending on how much code is likely to depend on it. For instance I'd spend more effort in the DOM than I would in the controller (remember this is degrees of effort not lots/none). There's also nothing to stop you doing refactoring _before_ you start on a piece of code. Maybe you didn't think it would be worth refactoring at the time but now you're adding a new feature you can see that applying the refactoring would make it easier.