Much, much worse is users having to deal with things randomly breaking after an update because someone decided they could make it better.
That's not to say you can't seek improvement. The public interface can be expanded without impacting existing uses. If, for example, an existing function doesn't reflect your current view of the world add a new one rather than try to jerry-rig the old one. If the new solutions are radically different such that you are essentially rewriting your code from scratch, a clean break is probably the better route to go.
If you are confident that existing users are no longer touching your legacy code, remove it rather than refactor it.
That's not to say you can't seek improvement. The public interface can be expanded without impacting existing uses. If, for example, an existing function doesn't reflect your current view of the world add a new one rather than try to jerry-rig the old one. If the new solutions are radically different such that you are essentially rewriting your code from scratch, a clean break is probably the better route to go.
If you are confident that existing users are no longer touching your legacy code, remove it rather than refactor it.