Indeed, LLMs cannot do truly novel thinking, and the laundering analogy is spot-on.

However they're able to do more than just regurgitating code, I can have them explain to me the underlying (mathematical or whatever) concept behind the code and write new code from scratch myself, with that knowledge.

Can/should this new code be considered as derivative work, if the underlying principles were already documented in literature?

They can regurgitate explanations as well as code. I'd strongly recommend doing actual research: you'll find better (less-distorted, better laid out, more complete) explanations.