Ohhh I didn't know that. For others trying it:

* Right-click > copy copies the image itself

* Highlight the image, then Right-click > copy gives you the alt text

Huh, that's surprising. I've never actually checked but always considered that they'd give the same result. Somewhat cumbersome, if one considers how few people know of this functionality though.