I guess you trade responsiveness for 'on-demand' data transfers. I would guess that the moment you click the button is too late to resume the loading.
Otoh the low-res thumbnail might be just enough to show as a (big) placeholder to bridge the (short) loading time to bring the image to a resolution that the user won't notice a difference.
I think, though I don't know how good the actual support is, that you can start the download from an arbitrary point so if you already have the thumbnail, you only need to download from that point on. If you are 80% sure people are going to click on the link to zoom, you could preload medium and then only load the final quality after they click.
One of the surprises of progressive loading, and why it's so good that JPEGXL has it, is how quickly you get to "good enough" and showing that before you get all the way to perfect.
