Suppose you have a service A, that uses two classes X and Y. Then you realize that it would be better for A to use X' and Y' (with different interfaces). If you have unit tests for X and Y, you will have to change these tests as well. If you do a lot of refactoring, you will spend a lot of time maintaining the unit tests. On the other hand, if you rely on integration tests (which test A), you still test X and Y, but your tests are less likely to change over time.

I'm not saying unit tests don't have their use, but as far as refactoring goes, I think they are a hindrance. Unless we talk about rewriting just a function.