This is central to Fowler's point, which is that technical debt only needs to be paid off if it gets "activated" a lot - this is where the metaphor breaks down, because typical debt accrues over time.
If you never touch code again, but that code was rushed and is ugly, there's no good reason to pay off the debt.
Similarly, if the code's getting thrown out (and if you design code to get thrown out), there's less reason to prioritize cleanup.
That's often overlooked, yeah - if it ain't broke, don't fix it. This goes against a lot of people that read code for the first time, myself included.
A recent example: React introduced hooks, which effectively means that there's a new de facto standard way to write components and you shouldn't be writing components as JS classes anymore.
The kneejerk reaction is then "We need to rewrite all our class-based components to use hooks!", which can take quite a long time (and it's tedious, boring busywork).
Luckily the React documentation on hooks itself [0] basically tells you to not bother unless you're actually going to work in that component anyway, and even then you should make sure everyone else in your team understands them; classes are fine, it'll keep working for a long time, resist your kneejerk reaction.
Yeah, I try to align tech debt cleanup with 'touch points', since that's generally an indicator that the code needs work done on it anyway.
It's sometimes controversial to say "We should not prioritize tech debt" but I feel pretty strongly that it's true. We aren't paid to write beautiful code, we're paid to write code that gets the job done. If code is getting the job done, even if it's fragile, our job is done. Only if that code is impeding new work should we prioritize it.
If you never touch code again, but that code was rushed and is ugly, there's no good reason to pay off the debt.
Similarly, if the code's getting thrown out (and if you design code to get thrown out), there's less reason to prioritize cleanup.