Maybe if you wrote the entire codebase yourself just a week ago so everything is still fresh in your memory.

In most real-world situations that is not the case, and refactoring becomes a careful task that leads to regressions even with testing in place.