I disagree. The reason those TODOs appear is because the answer to the question "Can you do it immediately?" is, "yes, I can, but it would break my flow and interrupt what I'm doing right now". The reason some of them don't later get promoted to issues in the tracker is because they're too small or otherwise irrelevant at the project level. Their place is in the code, where the next person looking at it will spot them.
Then it is not a TODO, it is a NOTE. Don't act as if it will be _done_ magically in the future. If I stumble upon a TODO, I won't fix it (because it will diasrupt my flow - the reason for the TODO in the first place). And if it is not in the issue tracker, I can't plan for it. A solution would be some kind of an "implement as many TODOs as possible" time-boxed spike, but this never worked really well for me in the past.
